A calibration image acquisition method, device, and storage medium

Abstract

The present disclosure relates to a calibration image acquisition method, apparatus, and storage medium. The method includes: obtaining a captured image obtained by an image acquisition device capturing a screen at a current capture moment; performing feature point detection on the captured image, and determining coverage information of the captured image based on the detected feature points; the coverage information includes a corresponding area covered by the feature points on the field of view of the image acquisition device and / or a corresponding area covered by the feature points on the screen; determining whether the captured image meets a preset requirement according to the coverage information of the captured image and the coverage information of a historical calibration image, and using the captured image as a calibration image when the captured image meets the preset requirement. Through the present disclosure, redundant data that is of low quality, repetitive, and does not contribute to the area covered on the field of view of the image acquisition device and / or the area covered on the screen is avoided, ensuring the quality of the acquired calibration images and improving the efficiency and stability of calibration image acquisition.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a calibration image acquisition method, apparatus, and storage medium. Background Art

[0002] In traditional film and television production, to meet shooting requirements, a large amount of time and manpower are required to select shooting locations, produce props, and build shooting scenes. However, virtual shooting (or virtual production) technology can use virtual scenes rendered by a rendering engine to replace real sets, reducing the dependence of film and television shooting on locations and sets, and greatly reducing shooting costs. At the same time, with the ability of real-time rendering, for some special effects that originally required post-production, virtual shooting can see the finished film effect during the shooting stage, realizing the front-loading of post-production and improving production efficiency. Due to the existence of these advantages, virtual shooting has been increasingly applied to film and television production in recent years.

[0003] In virtual shooting, to achieve a perfect fusion of "virtual" and "real", it is usually necessary to first determine the geometric model parameters of the camera through camera calibration technology, and the calibration accuracy determines the authenticity of virtual-real interaction in subsequent processes; and collecting calibration images is the first step in camera calibration, and the quality of calibration images directly affects the final calibration accuracy. Most of the existing calibration image acquisition methods collect calibration images by manually collecting single images, that is, the operator controls the movement and rotation of the camera (or calibration board), and at different positions and observation angles, collect images one by one as the final calibration images. During this process, how many images to collect, what kind of images to collect, and whether the collected images can be used for calibration are all determined by the operator, which is prone to redundant collected data, and it is difficult to ensure the quality and stability of the collected calibration images. Summary of the Invention

[0004] In view of this, the present disclosure provides a calibration image acquisition method, apparatus, electronic device, storage medium, and computer program product.

[0005] According to one aspect of the present disclosure, a calibration image acquisition method is provided, and the method includes:

[0006] Obtain a captured image obtained by an image acquisition device capturing a screen at a current capture moment, where an on-screen image including feature points is displayed on the screen;

[0007] Perform feature point detection on the captured image, and determine coverage information of the captured image based on the detected feature points; the coverage information includes a corresponding area covered by the feature points on the field of view of the image acquisition device and / or a corresponding area covered by the feature points on the screen;

[0008] Based on the coverage information of the acquired image and the coverage information of the historical calibrated image, determine whether the acquired image meets the preset requirements, and when the acquired image meets the preset requirements, use the acquired image as the calibrated image, where the historical calibrated image represents an image that meets the preset requirements acquired before the current acquisition time.

[0009] In a possible implementation, after using the acquired image as the calibrated image, it further includes:

[0010] Based on the coverage information of the acquired image and the coverage information of the historical calibrated image, determine the coverage rate at the current acquisition time, where the coverage rate represents the coverage degree of the acquired image and the historical calibrated image with respect to the screen and / or the coverage degree with respect to the field of view of the image acquisition device;

[0011] When the coverage rate exceeds the first preset threshold, prompt the user to terminate the current acquisition work, and use the acquired image and the historical calibrated image as images for calibrating the image acquisition device.

[0012] In a possible implementation, the field of view of the image acquisition device includes multiple preset regions, and the corresponding region covered by the feature point on the field of view of the image acquisition device is represented by the preset region where the feature point is located;

[0013] The preset requirements include:

[0014] At least one preset region where the feature point is located in the acquired image is different from all the preset regions where the feature point is located in the historical calibrated image.

[0015] In a possible implementation, the screen includes multiple boxes, and the corresponding region covered by the feature point on the screen is represented by the box where the feature point is located;

[0016] The preset requirements include:

[0017] At least one box where the feature point is located in the acquired image is different from all the boxes where the feature point is located in the historical calibrated image.

[0018] In a possible implementation, the method further includes:

[0019] Determine the acquisition time corresponding to each image in the historical calibrated image;

[0020] Select the target time adjacent to the current acquisition time from the acquisition times corresponding to each image;

[0021] The preset requirements further include:

[0022] All the boxes where the feature points are located in the acquired image have an intersection over union ratio less than a second preset threshold with all the boxes where the feature points are located in the image corresponding to the target moment;

[0023] Alternatively, the intersection over union ratio of all the preset regions where the feature points are located in the acquired image with all the preset regions where the feature points are located in the image corresponding to the target moment is less than a third preset threshold.

[0024] In a possible implementation manner, the method further includes:

[0025] Dividing the field of view of the image acquisition device into multiple preset regions;

[0026] Determining the coordinates of each feature point in the acquired image in the image coordinate system;

[0027] Determining the preset region where each feature point in the acquired image is located according to the coordinates of each feature point in the acquired image in the image coordinate system.

[0028] In a possible implementation manner, the density of the feature points included in the on-screen image is determined by the focal length of the image acquisition device, where the density of the feature points is positively correlated with the focal length.

[0029] In a possible implementation manner, the method further includes:

[0030] Determining the number of images in the historical calibration image;

[0031] When the number of images is greater than a preset number, prompting the user to terminate the current acquisition work.

[0032] According to another aspect of the present disclosure, there is provided a calibration image acquisition device, the device includes:

[0033] An acquisition module, configured to acquire an acquired image obtained by the image acquisition device shooting a screen at the current acquisition moment, where the screen displays an on-screen image including feature points;

[0034] A detection module, configured to perform feature point detection on the acquired image and determine the coverage information of the acquired image based on the detected feature points; the coverage information includes the corresponding region covered by the feature points on the field of view of the image acquisition device and / or the corresponding region covered by the feature points on the screen;

[0035] A judgment module, configured to judge whether the acquired image meets a preset requirement according to the coverage information of the acquired image and the coverage information of the historical calibration image, and use the acquired image as a calibration image when the acquired image meets the preset requirement, where the historical calibration image represents an image that meets the preset requirement acquired before the current acquisition moment.

[0036] In a possible implementation, the determination module is further configured to: determine the coverage rate at the current acquisition moment according to the coverage information of the acquired image and the coverage information of the historical calibrated image, where the coverage rate represents the coverage degree of the acquired image and the historical calibrated image with respect to the screen and / or the coverage degree with respect to the field of view of the image acquisition device; in the case where the coverage rate exceeds a first preset threshold, prompt the user to terminate the current acquisition work, and use the acquired image and the historical calibrated image as the images for calibrating the image acquisition device.

[0037] In a possible implementation, the field of view of the image acquisition device includes a plurality of preset regions, and the corresponding region covered by the feature points on the field of view of the image acquisition device is represented by the preset region where the feature points are located; the preset requirement includes: at least one preset region where the feature points are located in the acquired image is different from all the preset regions where the feature points are located in the historical calibrated image.

[0038] In a possible implementation, the screen includes a plurality of boxes, and the corresponding region covered by the feature points on the screen is represented by the box where the feature points are located; the preset requirement includes: at least one box where the feature points are located in the acquired image is different from all the boxes where the feature points are located in the historical calibrated image.

[0039] In a possible implementation, the determination module is further configured to: determine the acquisition moments corresponding to the respective images in the historical calibrated image; screen out the target moments adjacent to the current acquisition moment from the acquisition moments corresponding to the respective images; the preset requirement further includes: the intersection over union of all the boxes where the feature points are located in the acquired image and all the boxes where the feature points are located in the image corresponding to the target moment is less than a second preset threshold; or, the intersection over union of all the preset regions where the feature points are located in the acquired image and all the preset regions where the feature points are located in the image corresponding to the target moment is less than a third preset threshold.

[0040] In a possible implementation, the determination module is further configured to: divide the field of view of the image acquisition device into a plurality of preset regions; determine the coordinates of each feature point in the acquired image in the image coordinate system; and determine the preset region where each feature point in the acquired image is located according to the coordinates of each feature point in the acquired image in the image coordinate system.

[0041] In a possible implementation, the density of the feature points included in the image on the screen is determined by the focal length of the image acquisition device, where the density of the feature points is positively correlated with the focal length.

[0042] In a possible implementation, the determining module is further configured to: determine the number of images in the historical calibration image; and in the case where the number of images is greater than a preset number, prompt the user to terminate the current acquisition work.

[0043] According to another aspect of the present disclosure, there is provided an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0044] According to another aspect of the present disclosure, there is provided a computer-readable storage medium, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0045] According to another aspect of the present disclosure, there is provided a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, wherein when the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0046] Through the embodiments of the present disclosure, an acquisition image obtained by an image acquisition device capturing a screen at a current acquisition moment is acquired, and then feature point detection is performed on the acquisition image, and coverage information of the acquisition image is determined based on the detected feature points; the coverage information includes a corresponding area covered by the feature points on the field of view of the image acquisition device and / or a corresponding area covered by the feature points on the screen; according to the coverage information of the acquisition image and the coverage information of the historical calibration image, it is determined whether the acquisition image meets a preset requirement, and in the case where the acquisition image meets the preset requirement, the acquisition image is used as a calibration image. In some scenarios, during the process of a collector controlling the image acquisition device to capture a picture on the screen in real time or during the process of the image acquisition device capturing a picture on the screen according to a preset rule in real time, effective video frames can be screened out from the video stream in real time and automatically as the final calibration image; in other scenarios, after the collector controls the image acquisition device to capture a video stream and completes the data acquisition work, it is possible to judge the images already acquired by the collector, screen out effective video frames as the final calibration image, realize calibration image acquisition based on the video stream, avoid collecting low-quality, repeated, redundant data that does not contribute to the area covered on the field of view of the image acquisition device and / or the area covered on the screen, and ensure the quality of the acquired calibration images; at the same time, the frequency of manual operations and the randomness and instability brought by manual acquisition are effectively reduced, and the efficiency and stability of calibration image acquisition are improved; in addition, compared with the existing method of using a calibration board to acquire calibration images, the method in the embodiments of the present disclosure does not require all feature points to be covered in one acquisition, and only needs the acquisition image to meet the preset requirements, which is more convenient and efficient.

[0047] Other features and aspects of the present disclosure will become apparent from the following detailed description of the exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are included in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present disclosure, and together with the description serve to explain the principles of the present disclosure.

[0049] Figure 1 FIG. 1 shows a schematic diagram of a virtual shooting scene according to an embodiment of the present disclosure;

[0050] FIGS. 2(a)-(b) show schematic diagrams of a calibration plate image according to an embodiment of the present disclosure;

[0051] Figure 3 FIG. 3 shows a flowchart of a calibration image acquisition method according to an embodiment of the present disclosure;

[0052] FIGS. 4(a)-(b) show schematic diagrams of a screen and an image on the screen according to an embodiment of the present disclosure;

[0053] Figure 5 FIG. 5 shows a schematic diagram of a preset area according to an embodiment of the present disclosure;

[0054] Figure 6 FIG. 6 shows a flowchart of a calibration image acquisition method according to an embodiment of the present disclosure;

[0055] Figure 7 FIG. 7 shows a flowchart of a calibration image acquisition method according to an embodiment of the present disclosure;

[0056] Figure 8 FIG. 8 shows a structural diagram of a calibration image acquisition device according to an embodiment of the present disclosure;

[0057] Figure 9 FIG. 9 shows a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. Like reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0059] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present disclosure. Thus, statements such as "exemplary," "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., which appear at various places in this specification, are not necessarily all referring to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and their variants all mean "including but not limited to," unless otherwise specifically emphasized.

[0060] In the present disclosure, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects and indicates that there can be three relationships. For example, A and / or B can represent: including the case where A exists alone, the case where A and B exist simultaneously, and the case where B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (item)" or its similar expression refers to any combination of these items, including any combination of a single item or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, c can be single or multiple.

[0061] In addition, for a better illustration of the present disclosure, numerous specific details are given in the following detailed description. Those skilled in the art should understand that the present disclosure can be implemented without some of these specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0062] Figure 1 A schematic diagram of a virtual shooting scene according to an embodiment of the present disclosure is shown, as Figure 1 shown, the virtual shooting scene may include a display device 10 and an image acquisition device 20; wherein, the display device 10 is used to display the rendered scene, and the image acquisition device 20 is used to capture the image displayed by the display device 10. For example, an actor can stand at an appropriate position in front of the display device 10 and perform with the display device 10 as the background, and the image acquisition device 20 can simultaneously capture the actor and the image displayed by the display device 10, thereby completing the shooting of the virtual scene.

[0063] Exemplarily, the display device 10 may be a screen made of LED (Light-Emitting Diode) material or a screen made of other materials, and is used to display various information such as text, images, videos, etc.; the shape of the screen may be a flat screen, a curved screen, a triple-fold screen, or a special-shaped screen with a multi-faceted three-dimensional structure, etc., and the embodiments of the present disclosure do not limit this. Among them, the screen made of LED material may be composed of one or more LED sub-screens, and each LED sub-screen may be spliced by a plurality of LED boxes. One side of the LED box is an LED panel. By splicing a plurality of LED boxes in a row and column manner, a large-sized LED sub-screen composed of the LED panels on each LED box can be obtained; according to the different angles between adjacent LED panels, a flat screen, a curved screen, a triple-fold screen, or a special-shaped screen can be formed. When the LED sub-screen includes a plurality of LED boxes, the LED sub-screen may be spliced by LED boxes of one model, or may be spliced by LED boxes of multiple models, and LED boxes of different models may have different resolutions and / or sizes.

[0064] Exemplarily, the image acquisition device 20 may be a device with a shooting function such as a camera or a video camera. The image acquisition device 20 is equipped with a zoom lens, a fixed-focus lens, or a binocular lens, etc., so as to meet different shooting requirements.

[0065] In the process of virtual shooting using the image acquisition device 20, the geometric model parameters of the image acquisition device 20 are crucial. Therefore, before using the image acquisition device 20 for virtual shooting, it is necessary to calibrate the image acquisition device 20, that is, to solve the geometric model parameters of the image acquisition device 20. Exemplarily, the geometric model parameters of the image acquisition device 20 may include internal parameters and / or external parameters. Among them, the external parameters may include pose, that is, the position of the lens of the image acquisition device 20 in space and the orientation of the lens, which can be represented by a rotation matrix R and a translation matrix T; the internal parameters may include the resolution of the lens of the image acquisition device 20, the field of view (FOV), the position of the principal point, the focal length, distortion parameters (such as radial distortion, tangential distortion), etc.

[0066] In the existing calibration image acquisition method, the image of a calibration board (a flat plate with a fixed-spacing pattern array) displayed on the screen is usually acquired. For example, FIGS. 2(a)-(b) show schematic diagrams of calibration board images according to an embodiment of the present disclosure. FIG. 2(a) is a checkerboard pattern, which consists of black and white square grids. The points formed at the junctions of two adjacent and same-color square grids are called corner points; FIG. 2(b) is a dot matrix pattern, which consists of multiple black dots on a white background. Whether it is a calibration board image of the checkerboard type or a calibration board image of the grid point type, the calibration images are collected by manually acquiring single images, that is, the operator controls the movement and rotation of the camera (or calibration board), and at different positions and different observation angles, single images are acquired as the final calibration images. In this process, how many images are acquired, what kind of images are acquired, and whether the acquired images can be used for calibration are all determined by the operator, which is prone to data redundancy in acquisition, and it is difficult to ensure the quality and stability of the acquired calibration images.

[0067] To solve the above technical problems, an embodiment of the present disclosure proposes a calibration image acquisition method based on a video stream (for specific description, see below), which can be used to acquire calibration images for calibrating the image acquisition device 20 in the virtual shooting scene shown above Figure 1 It can automatically screen out effective video frames from the video stream as the final calibration images, avoiding the acquisition of redundant data such as low-quality, repeated, or data that does not contribute to the coverage information of the image, ensuring the quality of the acquired calibration images. In addition, it effectively reduces the frequency of manual operations and the randomness and instability brought by manual acquisition, and improves the efficiency and stability of calibration image acquisition. As an example, it can also automatically determine whether the acquired calibration images are sufficient to complete the calibration and prompt the user in a timely manner, avoiding the uncertainty brought by introducing manual judgment methods, and effectively improving the success rate of subsequent calibration and the stability of the calibration effect.

[0068] It should be noted that the above-described application scenarios described in the embodiments of the present disclosure are for more clearly explaining the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those of ordinary skill in the art know that for the emergence of other similar or new scenarios, for example, the acquisition of calibration images for other shooting scenarios involving cameras, etc., the technical solutions provided by the embodiments of the present disclosure are equally applicable to similar technical problems.

[0069] The calibration image acquisition method provided by the embodiments of the present disclosure will be described in detail below.

[0070] Figure 3 FIG. shows a flowchart of a calibration image acquisition method according to an embodiment of the present disclosure. This method can be executed by an electronic device with data processing functions such as a processor or a server; asFigure 3 As shown, the method may include the following steps:

[0071] Step 301: Obtain a captured image obtained by an image capturing device capturing a screen at the current capture moment, where the screen displays an on-screen image containing feature points.

[0072] For example, the image capturing device may be the image capturing device 20 described above Figure 1 and the screen may be the image display device 10 described above Figure 1 in the above.

[0073] In some scenarios, during the process of a collector controlling an image capturing device to capture an on-screen image on the screen in real time, or during the process of the image capturing device capturing an on-screen image on the screen in real time according to a preset rule (such as a moving direction, a moving speed, a preset route, etc.), video frames can be extracted from the captured video frames at equal intervals in real time. For example, starting from the start of collection, every 0.3 seconds, a currently captured video frame can be extracted as the captured image obtained by the image capturing device capturing the screen at the current capture moment. That is, in this scenario, the current capture moment is the moment of capturing video frames in real time. In other scenarios, after a collector controls an image capturing device to capture a video stream and completes the data collection work, video frames in the already captured video stream can be extracted at equal intervals in sequence, and each extracted video frame is used as the captured image obtained by the image capturing device capturing the screen at the current capture moment. That is, in this scenario, the current capture moment is the moment when the video frames were previously captured.

[0074] Exemplarily, before the image capturing device captures the screen, an on-screen image containing feature points can be generated and projected onto the screen. Triggered manually, a video stream collection starts. After the collection starts, the on-screen image will always be displayed on the screen. This on-screen image can also be called a calibration board image, and the pose of the image capturing device is changed during the collection process.

[0075] Among them, the feature points represent points with a drastic change in grayscale value in the on-screen image or points with a large curvature on the edge of the on-screen image. These points can reflect the essential features of the image and can identify information such as objects or positions in the on-screen image. Exemplarily, the on-screen image may include one or more preset feature points. For example, the on-screen image may be the checkerboard image shown in Fig. 2(a), and the corner points formed at the joints of two adjacent and same-color square grids are feature points. It may also be the dot matrix image shown in Fig. 2(b), where the black dots on the white background are feature points.

[0076] Exemplarily, each feature point included in the upper screen image can be numbered in advance, and the numbers of different feature points are different; it is also possible to record the cabinet to which each feature point belongs. For example, FIGS. 4(a)-(b) show schematic diagrams of a screen and an upper screen image according to an embodiment of the present disclosure. Among them, FIG. 4(a) is an LED screen composed of three sub-screens, and each sub-screen is composed of nine cabinets arranged in a three-row and three-column manner; FIG. 4(b) is an upper screen image projected on the three sub-screens. The upper screen image contains multiple feature points. Among the three sub-screens, each cabinet of the first sub-screen displays one feature point, each cabinet of the second sub-screen displays two feature points, and each cabinet of the third sub-screen displays three feature points, and each feature point is numbered, and the numbers of different feature points are different.

[0077] In a possible implementation manner, the information of the screen can be obtained in advance, including: the number of sub-screens, the number of rows and columns of the cabinets of each sub-screen, the resolution of the cabinets or the size of the cabinets, etc. Then, an upper screen image suitable for being displayed on each cabinet can be generated according to the screen information, so as to ensure that the feature points in the upper screen image can be normally displayed on the corresponding cabinets. Exemplarily, when the screen includes multiple sub-screens, the corresponding upper screen sub-images can be generated respectively for each sub-screen, and the upper screen sub-images displayed on all sub-screens form the upper screen image. The number, type (such as shape, color, size, etc.), arrangement method, etc. of the feature points included in the upper screen sub-images displayed in different sub-screens can be the same or different. Exemplarily, when a sub-screen includes multiple cabinets, the corresponding upper screen sub-images can be generated respectively for each cabinet, and the upper screen sub-images displayed on all cabinets of each sub-screen form the upper screen image. The number, type, arrangement method, etc. of the feature points included in the upper screen sub-images displayed in different cabinets can be the same or different.

[0078] In a possible implementation, the density of feature points included in the on-screen image is determined by the focal length of the image acquisition device, where the density of feature points is positively correlated with the focal length. That is, the larger the focal length of the image acquisition device, the greater the density of feature points; conversely, the smaller the focal length of the image acquisition device, the smaller the density of feature points. It can be understood that when the relative distance between the image acquisition device and the screen remains unchanged, the larger the focal length of the image acquisition device, the smaller the shooting field of view of the image acquisition device, and the less effective part of the on-screen image captured by the image acquisition device. At this time, the density of feature points included in the on-screen image can be configured to a larger value, so as to ensure that the captured image obtained by the image acquisition device shooting the screen at a larger focal length still contains feature points for subsequent feature point detection. And the smaller the focal length of the image acquisition device, the larger the shooting field of view of the image acquisition device, and the more effective part of the on-screen image captured by the image acquisition device. At this time, the density of feature points included in the on-screen image can be configured to a smaller value, so as to avoid excessive operation processing on the detected feature points and improve the processing efficiency.

[0079] Step 302: Perform feature point detection on the captured image, and determine the coverage information of the captured image based on the detected feature points; the coverage information includes the corresponding area covered by the feature points in the field of view of the image acquisition device and / or the corresponding area covered by the feature points on the screen.

[0080] It can be understood that at any moment, the image acquisition device may capture part or all of the on-screen image, so the image acquisition device may capture one or more feature points in the on-screen image, that is, the captured image obtained may correspondingly include one or more feature points. Performing feature point detection on the captured image can detect the one or more feature points; in addition, the part of the on-screen image captured by the image acquisition device has no feature points, that is, the captured image obtained does not include feature points. Performing feature point detection on the captured image, the detected feature points are empty, and correspondingly, the determined coverage information of the captured image is empty.

[0081] Exemplarily, various existing feature point detection algorithms can be used to perform feature point detection on the captured image, such as: circle detection algorithm, ellipse detection algorithm, or blob detection algorithm, etc. Exemplarily, the feature point detection algorithm adopted can be determined according to the type of feature points included in the on-screen image to improve the detection accuracy; for example, if the on-screen image is a checkerboard image shown in Fig. 2(a), the blob detection algorithm can be adopted, and if the on-screen image is a dot matrix image of circles shown in Fig. 2(b), the circle detection algorithm can be adopted. Exemplarily, each feature point can also be numbered according to a preset, the numbers of the detected feature points can be determined, and the coordinates of the detected feature points in the image coordinate system (i.e., the coordinate system of the captured image) can also be determined.

[0082] In some scenarios, the coverage information of the captured image includes the corresponding area covered by the detected feature points on the field of view of the image capture device; after performing feature point detection on the captured image, the corresponding area covered by each detected feature point on the field of view of the image capture device can be determined based on the detected feature points, and the corresponding areas covered by all the detected feature points on the field of view of the image capture device can represent the coverage information of the captured image.

[0083] As an example, the field of view of the image capture device includes multiple preset areas, and the corresponding area covered by the feature points on the field of view of the image capture device is represented by the preset area where the feature points are located. Among them, the sizes and shapes of different preset areas can be the same or different, and no limitation is made in this regard; for example, Figure 5 A schematic diagram of a preset area according to an embodiment of the present disclosure is shown, as Figure 5 shown, the field of view of the image capture device includes four preset areas of the same size, namely area 1, area 2, area 3, and area 4 in the figure; it is detected that the captured image obtained by photographing the screen at the current capture moment contains three feature points, namely feature point a, feature point b, and feature point c in the figure, where feature point a is located in area 3, and feature point b and feature point c are located in area 4, then the coverage information of the captured image can be determined as area 3 and area 4.

[0084] In a possible implementation manner, the method further includes: dividing the field of view of the image capture device into multiple preset areas; determining the coordinates of each feature point in the captured image in the image coordinate system; and determining the preset area where each feature point in the captured image is located according to the coordinates of each feature point in the captured image in the image coordinate system.

[0085] Exemplarily, according to the resolution of the captured image, the field of view of the image capture device can be divided into M*N rectangular areas, where M and N are both integers not less than 1, and the specific values of M and N can be set according to requirements, and no limitation is made in this regard; for example, if the resolution of the captured image is 1080*720, M = N = 10 can be set, that is, the field of view of the image capture device can be divided into 100 rectangular areas of the same size, and the size of each rectangular area is 108*72.

[0086] Exemplarily, the rectangular areas can also be numbered in sequence, and the numbers of different rectangular areas are different. For the rectangular areas shown above Figure 5 as shown, taking the coordinate origin of the image coordinate system as the base point, and following the principle of the positive X-axis direction first and then the positive Y-axis direction, the rectangular areas can be numbered in sequence, so that the four preset areas divided from the field of view of the image capture device are numbered as area 1, area 2, area 3, and area 4.

[0087] Exemplarily, when detecting feature points in the acquired image, the coordinates of each detected feature point in the image coordinate system can be determined. Furthermore, based on the coordinates of each feature point in the image coordinate system, the preset region where each feature point in the acquired image is located can be determined. It can be understood that for any point in the image coordinate system, this point is located within a preset region. For example, if the coordinates of a certain detected feature point in the image coordinate system are (x, y), the resolution of the acquired image is W*H, and the field of view of the image acquisition device is divided into M*N rectangular regions, then the rectangular region where this feature point is located is Block_id = Floor(x / (W / N))*N + Floor(y / (H / M)), where Block_id is the number of the rectangular region, and Floor() represents rounding down. In this way, by traversing all the detected feature points, the rectangular regions where each feature point in the acquired image is located can be obtained, thereby obtaining the coverage information of the acquired image.

[0088] In some other scenarios, the coverage information of the acquired image includes the corresponding regions covered by the detected feature points on the screen. After detecting the feature points in the acquired image, based on the detected feature points, the corresponding region covered by each detected feature point on the screen can be determined. Detecting the corresponding regions covered by all the feature points on the screen can represent the coverage information of the acquired image.

[0089] As another example, the screen includes multiple boxes, and the corresponding region covered by the feature point on the screen is represented by the box where the feature point is located. Exemplarily, the boxes constituting the screen can be numbered in sequence from top to bottom and from left to right, and the numbers of different boxes are different. For example, in the above Figure 4(a), box 1, box 2...; as mentioned before, when generating the on-screen image containing the feature points, the box to which each feature point belongs can be recorded. For example, the box number where each feature point is located can be recorded. In this way, when detecting the feature points in the acquired image, the boxes where each detected feature point is located can be determined, and by traversing all the detected feature points, the coverage information of the acquired image can be obtained.

[0090] Step 303: According to the coverage information of the acquired image and the coverage information of the historical calibrated image, determine whether the acquired image meets the preset requirements. If the acquired image meets the preset requirements, use the acquired image as the calibrated image, where the historical calibrated image represents an image that meets the preset requirements acquired before the current acquisition time.

[0091] Exemplarily, in the case where the acquired image obtained by photographing the screen at the current acquisition time does not meet the preset requirements, the acquired image will be discarded, and a new acquired image will be continuously obtained as the image obtained by photographing the screen at the current acquisition time, and the above steps 301 - 303 will be repeatedly executed.

[0092] It can be understood that when collecting calibration images, it is generally necessary to continuously collect a video stream of a certain duration, extract video frames from the video frames at equal intervals, and use them as the collected images obtained by shooting the screen at the current collection moment in sequence. Then, perform the above steps 301-303 to determine whether the extracted video frames are valid video frames. The historical calibration images are the valid video frames that have been screened out before the current collection moment during this collection process, that is, the images that meet the preset requirements collected before the current collection moment. Among them, for the first extracted video frame, the corresponding historical calibration image is empty.

[0093] As an example, the coverage information of the collected image obtained by shooting the screen at the current collection moment includes the preset area where the feature points are located in the collected image; the coverage information of the historical calibration image includes the preset area where the feature points are located in the historical calibration image; the preset requirements include: at least one preset area where the feature points are located in the collected image is different from all the preset areas where the feature points are located in the historical calibration image. In this way, the collected image obtained by shooting the screen at the current collection moment can be comprehensively compared with all the images that meet the preset requirements collected before the current collection moment. When at least one preset area where the feature points are located in the collected image obtained by shooting the screen at the current collection moment is different from all the preset areas where the feature points are located in the historical calibration image, it indicates that the feature points collected in the collected image obtained by shooting the screen at the current collection moment cover a new field of view area of the image acquisition device. For example, for the above Figure 5 As shown, feature point a is located in area 3, and feature points b and c are located in area 4. If all the rectangular areas where the detected feature points are located in the historical calibration image are areas 1, 2, and 3, that is, all the feature points in the historical calibration image are not located in area 4, it indicates that the feature points collected in the collected image obtained by shooting the screen at the current collection moment cover the new field of view area 4 of the image acquisition device. Thus, it can be determined that the collected image obtained by shooting the screen at the current collection moment meets the preset requirements, and the collected image obtained by shooting the screen at the current collection moment can be used as the calibration image. If all the rectangular areas where the detected feature points are located in the historical calibration image are areas 2, 3, and 4, it indicates that the feature points collected in the collected image obtained by shooting the screen at the current collection moment do not cover the new field of view area of the image acquisition device. Thus, it can be determined that the collected image obtained by shooting the screen at the current collection moment does not meet the preset requirements.

[0094] As another example, the coverage information of the captured image obtained by capturing the screen at the current capture moment includes the box where the feature points in the captured image are located; the coverage information of the historical calibration image includes the box where the feature points in the historical calibration image are located; the preset requirements include: at least one box where the feature points in the captured image are located is different from all the boxes where the feature points in the historical calibration image are located. In this way, the captured image obtained by capturing the screen at the current capture moment can be comprehensively compared with all the images that meet the preset requirements captured before the current capture moment. When at least one box where the feature points in the captured image obtained by capturing the screen at the current capture moment are located is different from all the boxes where the feature points in the historical calibration image are located, it indicates that the feature points captured in the captured image obtained by capturing the screen at the current capture moment cover new boxes on the screen. For example, if in the above Figure 5 the boxes where feature points a, b, and c are located are all box 1 in Fig. 4(b), and the boxes where the detected feature points in the historical calibration image are located are box 2 and box 3, that is, all the feature points in the historical calibration image are not located in box 1, it indicates that the feature points captured in the captured image obtained by capturing the screen at the current capture moment cover the new box 1 on the screen. Then, the captured image obtained by capturing the screen at the current capture moment meets the preset requirements and can be used as the calibration image; if in the above Figure 5 the boxes where feature points a, b, and c are located are all box 1 in Fig. 4(b), and the boxes where the detected feature points in the historical calibration image include box 1, it indicates that the feature points captured in the captured image obtained by capturing the screen at the current capture moment do not cover new boxes on the screen, so it can be determined that the captured image obtained by capturing the screen at the current capture moment does not meet the preset requirements.

[0095] As another example, the coverage information of the captured image obtained by capturing the screen at the current capture moment includes the boxes where the feature points are located in the captured image; the coverage information of the historical calibration image includes the boxes where the feature points are located in the historical calibration image, and the preset requirements include: the intersection over union of all the boxes where the feature points are located in the captured image and all the boxes where the feature points are located in the image corresponding to the target moment is less than a second preset threshold. Exemplarily, the capture moment corresponding to each image in the historical calibration image can be determined; the target moment adjacent to the current capture moment can be selected from the capture moments corresponding to each image; and then the intersection over union of all the boxes where the feature points are located in the captured image and all the boxes where the feature points are located in the image corresponding to the target moment can be calculated. In this way, the captured image obtained by capturing the screen at the current capture moment can be compared with the image corresponding to the target moment. When the intersection over union of all the boxes where the feature points are located in the captured image obtained by capturing the screen at the current capture moment and all the boxes where the feature points are located in the image corresponding to the target moment is less than the second preset threshold, it indicates that the feature points captured in the captured image obtained by capturing the screen at the current capture moment probably cover new boxes on the screen relative to the feature points in the image corresponding to the target moment. Exemplarily, the second preset threshold can be 60%. If the intersection over union of all the boxes where the feature points are located in the captured image obtained by capturing the screen at the current capture moment and all the boxes where the feature points are located in the image corresponding to the target moment is less than 60%, then the captured image obtained by capturing the screen at the current capture moment meets the preset requirements. If the intersection over union of all the boxes where the feature points are located in the captured image obtained by capturing the screen at the current capture moment and all the boxes where the feature points are located in the image corresponding to the target moment is not less than 60%, then the captured image obtained by capturing the screen at the current capture moment does not meet the preset requirements.

[0096] As another example, the coverage information of the captured image obtained by capturing the screen at the current capture moment includes the preset area where the feature points are located in the captured image; the coverage information of the historical calibration image includes the preset area where the feature points are located in the historical calibration image; the preset requirement includes: the intersection-over-union ratio of all the preset areas where the feature points are located in the captured image and all the preset areas where the feature points are located in the image corresponding to the target moment is less than a third preset threshold. In this way, the captured image obtained by capturing the screen at the current capture moment can be compared with the image corresponding to the target moment. When the intersection-over-union ratio of all the preset areas where the feature points are located in the captured image obtained by capturing the screen at the current capture moment and all the preset areas where the feature points are located in the image corresponding to the target moment is less than the third preset threshold, it indicates that the feature points captured in the captured image obtained by capturing the screen at the current capture moment, relative to the feature points in the image corresponding to the target moment, probably cover a new field of view area of the image capture device. Exemplarily, the third preset threshold can be 60%. If the intersection-over-union ratio of all the preset areas where the feature points are located in the captured image obtained by capturing the screen at the current capture moment and all the preset areas where the feature points are located in the image corresponding to the target moment is less than 60%, it is determined that the captured image obtained by capturing the screen at the current capture moment meets the preset requirement. If the intersection-over-union ratio of all the preset areas where the feature points are located in the captured image obtained by capturing the screen at the current capture moment and all the preset areas where the feature points are located in the image corresponding to the target moment is not less than 60%, it is determined that the captured image obtained by capturing the screen at the current capture moment does not meet the preset requirement.

[0097] In this way, by determining whether the acquired image obtained by capturing the screen at the current acquisition moment meets the preset requirements, it is determined whether the acquired image obtained by capturing the screen at the current acquisition moment can be used as a calibration image. The preset requirements include one or more of the following: at least one preset area where the feature points are located in the acquired image is different from all the preset areas where the feature points are located in the historical calibration image; at least one box where the feature points are located in the acquired image is different from all the boxes where the feature points are located in the historical calibration image; the intersection over union of all the boxes where the feature points are located in the acquired image and all the boxes where the feature points are located in the image corresponding to the target moment is less than a second preset threshold; the intersection over union of all the preset areas where the feature points are located in the acquired image and all the preset areas where the feature points are located in the image corresponding to the target moment is less than a third preset threshold. As an example, the preset requirements may include one of the following: at least one preset area where the feature points are located in the acquired image is different from all the preset areas where the feature points are located in the historical calibration image, and at least one box where the feature points are located in the acquired image is different from all the boxes where the feature points are located in the historical calibration image. It also includes one of the following: the intersection over union of all the boxes where the feature points are located in the acquired image and all the boxes where the feature points are located in the image corresponding to the target moment is less than the second preset threshold, and the intersection over union of all the preset areas where the feature points are located in the acquired image and all the preset areas where the feature points are located in the image corresponding to the target moment is less than the third preset threshold. Then, it is considered that the acquired image is a valid frame, and the acquired image is used as the calibration image.

[0098] In the embodiments of the present disclosure, an acquisition image obtained by an image acquisition device capturing a screen at the current acquisition moment is acquired, and then feature point detection is performed on the acquisition image, and coverage information of the acquisition image is determined based on the detected feature points; the coverage information includes a corresponding area covered by the feature points on the field of view of the image acquisition device and / or a corresponding area covered by the feature points on the screen; according to the coverage information of the acquisition image and the coverage information of the historical calibration image, it is determined whether the acquisition image meets a preset requirement, and when the acquisition image meets the preset requirement, the acquisition image is used as a calibration image. In some scenarios, during the process of a collector controlling the image acquisition device to capture the screen in real time or during the process of the image acquisition device capturing the screen in real time according to a preset rule, by performing the above steps 301-303, effective video frames can be screened out from the video stream in real time and automatically as the final calibration image; in other scenarios, after the collector controls the image acquisition device to capture the video stream and completes the data acquisition work, by performing the above steps 301-303, it is possible to judge the images previously acquired by the collector, screen out effective video frames as the final calibration image, realize the acquisition of calibration images based on the video stream, avoid collecting redundant data that is of low quality, repetitive, and has no contribution to the area covered on the field of view of the image acquisition device and / or the area covered on the screen, and ensure the quality of the acquired calibration images; in addition, the frequency of manual operations and the randomness and instability brought by manual acquisition are effectively reduced, and the efficiency and stability of calibration image acquisition are improved. In addition, compared with the existing method of using a calibration board to acquire calibration images, which requires covering all feature points in one acquisition, the method in the embodiments of the present disclosure does not require covering all feature points in one acquisition, and only needs the acquisition image to meet the preset requirement, which is more convenient and efficient.

[0099] Further, considering the existing calibration image acquisition method, the number of acquired calibration images (that is, when to end the acquisition work and start calibration) is completely determined by the operator. This method of manually judging whether the acquired calibration images are sufficient for calibration increases the uncertainty of the subsequent calibration effect. Therefore, after determining whether the acquisition image at the current acquisition moment can be used as a calibration image, in the embodiments of the present disclosure, it can be further determined whether the acquisition termination condition is reached.

[0100] Figure 6 The flowchart of a calibration image acquisition method according to an embodiment of the present disclosure is shown. As Figure 6 shown, the method may include the following steps:

[0101] Step 601, acquire an acquisition image obtained by an image acquisition device capturing a screen at the current acquisition moment, and an on-screen image including feature points is displayed on the screen.

[0102] This step is the same as the aboveFigure 3 This is the same as step 301 above and will not be elaborated here.

[0103] Step 602: Detect feature points in the acquired image, and determine the coverage information of the acquired image based on the detected feature points; the coverage information includes the corresponding area covered by the feature points on the field of view of the image acquisition device and / or the corresponding area covered by the feature points on the screen.

[0104] This step is the same as the above Figure 3 step 302 and will not be elaborated here.

[0105] Step 603: Determine whether the acquired image meets the preset requirements according to the coverage information of the acquired image and the coverage information of the historical calibration image, and use the acquired image as the calibration image when the acquired image meets the preset requirements.

[0106] This step is the same as the above Figure 3 step 303 and will not be elaborated here.

[0107] After using the acquired image as the calibration image, it further includes: determining the coverage rate at the current acquisition moment according to the coverage information of the acquired image and the coverage information of the historical calibration image, where the coverage rate represents the coverage degree of the acquired image and the historical calibration image with respect to the screen and / or the field of view of the image acquisition device.

[0108] As an example, the coverage rate can represent the coverage degree of the acquired image obtained by photographing the screen at the current acquisition moment and the historical calibration image with respect to the screen; exemplarily, the coverage degree of the acquired image and the historical calibration image with respect to the screen can be determined by the corresponding area covered by the feature points on the screen in the acquired image and the corresponding area covered by the feature points on the screen in the historical calibration image; among them, the corresponding area covered by the feature points on the screen can be represented by the box in the screen where the feature points are located. In this way, by counting the boxes where the feature points are located in the acquired image obtained by photographing the screen at the current acquisition moment and the boxes where the feature points are located in the historical calibration image, the total number of covered boxes is determined, and then the total number of covered boxes is compared with the total number of boxes constituting the screen to obtain the coverage rate at the current acquisition moment.

[0109] As another example, the coverage rate can represent the coverage degrees of the captured image obtained by capturing the screen at the current capture moment and the historical calibration image with respect to the field of view of the image capture device; exemplarily, the corresponding areas covered by the feature points in the captured image on the screen and the corresponding areas covered by the feature points in the historical calibration image on the screen can be collected to determine the coverage degrees of the captured image and the historical calibration image with respect to the screen; wherein, the corresponding areas covered by the feature points on the screen can be represented by a preset area in the field of view of the image capture device where the feature points are located. In this way, by counting the preset areas where the feature points are located in the captured image obtained by capturing the screen at the current capture moment and the preset areas where the feature points are located in the historical calibration image, the total number of covered preset areas is determined, and then the total number of covered preset areas is compared with the total number of preset areas constituting the field of view of the image capture device to obtain the coverage rate at the current capture moment.

[0110] As another example, the coverage rate can represent the coverage degrees of the captured image obtained by capturing the screen at the current capture moment and the historical calibration image with respect to the screen and with respect to the field of view of the image capture device. Exemplarily, the coverage rate at the current capture moment can include a first coverage rate and a second coverage rate. The first coverage rate can be obtained by comparing the total number of covered boxes with the total number of boxes constituting the screen in the manner of the foregoing example, and the second coverage rate can be obtained by comparing the total number of covered preset areas with the total number of preset areas constituting the field of view of the image capture device.

[0111] Step 605: When the coverage rate exceeds a first preset threshold, prompt the user to terminate the current capture work, and use the captured image and the historical calibration image as the images for calibrating the image capture device.

[0112] Among them, the first preset threshold can be set according to requirements; exemplarily, the first preset threshold can be 90%. If the coverage rate at the current capture moment exceeds 90%, it indicates that sufficient calibration images have been captured for calibrating the image capture device, and the user will be automatically prompted to terminate the current capture work. As an example, when both the first coverage rate and the second coverage rate at the current capture moment exceed 90%, it indicates that the captured calibration images can basically cover the entire field of view of the image capture device and can basically cover all the boxes on the screen, meeting the calibration requirements, and the user will be prompted to terminate the current capture work.

[0113] Exemplarily, when the coverage rate at the current capture moment does not exceed the first preset threshold, continue the capture work, use the captured image obtained by capturing the screen at the latest moment as the captured image at the current capture moment, and repeat the above steps 601 - 605.

[0114] In the embodiments of the present disclosure, after determining that the captured image at the current capture moment can be used as a calibration image, the coverage rate at the current capture moment is determined, and the user is prompted to terminate the current capture work when the coverage rate exceeds the first preset threshold; thus, it is automatically determined whether the captured calibration images are sufficient to complete the calibration and the user is timely prompted to end the current capture work. Compared with the existing method of manually determining whether the captured calibration images are sufficient for calibration, the subsequent calibration effect is more controllable and stable.

[0115] Figure 7 FIG. 4 shows a flowchart of a calibration image acquisition method according to an embodiment of the present disclosure. As Figure 7 shown, the method may include the following steps:

[0116] Step 801, obtain a captured image obtained by an image acquisition device by photographing a screen at the current capture moment, where an on-screen image including feature points is displayed on the screen.

[0117] This step is the same as step 301 above Figure 3 and will not be described in detail here.

[0118] Step 802, perform feature point detection on the captured image, and determine the coverage information of the captured image based on the detected feature points; the coverage information includes the corresponding area covered by the feature points on the field of view of the image acquisition device and / or the corresponding area covered by the feature points on the screen.

[0119] This step is the same as step 302 above Figure 3 and will not be described in detail here.

[0120] Step 803, determine whether the captured image meets the preset requirements according to the coverage information of the captured image and the coverage information of the historical calibration images, and use the captured image as a calibration image when the captured image meets the preset requirements.

[0121] This step is the same as step 303 above Figure 3 and will not be described in detail here.

[0122] Exemplarily, the coverage rate at the current capture moment can be determined, and when the coverage rate at the current capture moment does not exceed the first preset threshold, the following steps 804-805 are executed.

[0123] Step 804, determine the number of images in the historical calibration images.

[0124] Step 805, when the number of images is greater than the preset number, prompt the user to terminate the current capture work.

[0125] Among them, the preset quantity can be set according to requirements. It can be understood that during this acquisition process, when the number of acquired images is greater than the preset quantity and the acquired calibration images are not yet sufficient for calibration, it indicates that the quality of the images acquired at most times during this acquisition process is poor, and there may be reasons such as incorrect acquisition methods. If the acquisition continues, it will take more time and cost to possibly acquire calibration images that meet the requirements. Therefore, the user can be promptly prompted to terminate this acquisition work. Further, the user can be prompted to change the acquisition pose, acquisition speed, etc., and perform an acquisition again.

[0126] In the embodiments of the present disclosure, after determining that the acquired image at the current acquisition moment can be used as a calibration image, the number of images in the historical calibration images is determined, and when the number of images is greater than the preset quantity, the user is prompted to terminate this acquisition work; thereby realizing automatic judgment of the user's acquisition status and promptly prompting the user, effectively avoiding wasting acquisition time and cost.

[0127] In this way, through the above Figure 6 or Figure 7 It can be seen that after determining whether the acquired image at the current acquisition moment can be used as a calibration image, it can be further determined whether the preset termination condition is met, and when the preset termination condition is met, the user is prompted to terminate this acquisition work; among them, the preset termination condition can include one or more of the following: a. The coverage rate at the current acquisition moment exceeds the first preset threshold, b. The number of images in the historical calibration images is greater than the preset quantity; among them, when the preset termination condition b is met, the status of this acquisition can also be recorded as insufficient acquisition; when the preset termination condition a is met, the status of this acquisition can also be recorded as sufficient acquisition, that is, the acquired calibration images are sufficient to complete calibration. If the preset termination condition is not met, the acquisition work continues, and the acquired image at the latest moment is used as the acquired image at the current acquisition moment, and the steps shown in the above Figure 6 or Figure 7 are repeated. In this way, it is possible to determine whether this acquisition is sufficient based on the coverage rate at the current acquisition moment and the number of images in the historical calibration images, thereby ensuring that the acquired calibration images can be stably used for calibration of the image acquisition device, with strong scalability, avoiding the uncertainty brought by introducing manual judgment methods, and effectively improving the success rate of subsequent calibration and the stability of the calibration effect.

[0128] Based on the same inventive concept as the above method embodiments, the embodiments of the present disclosure further provide a calibration image acquisition device, which can be used to execute the technical solutions described in the above method embodiments. For example, it can execute the steps of the calibration image acquisition method shown in the above Figure 3 , Figure 6 or Figure 7 .

[0129] Figure 8 The structural diagram of a calibration image acquisition device according to an embodiment of the present disclosure is shown. As Figure 8 shown, the device may include: an acquisition module 901, configured to acquire an acquisition image obtained by the image acquisition device capturing a screen at the current acquisition moment, where the screen displays an on-screen image including feature points; a detection module 902, configured to perform feature point detection on the acquisition image and determine coverage information of the acquisition image based on the detected feature points; the coverage information includes a corresponding area covered by the feature points on the field of view of the image acquisition device and / or a corresponding area covered by the feature points on the screen; a judgment module 903, configured to judge whether the acquisition image meets a preset requirement according to the coverage information of the acquisition image and the coverage information of a historical calibration image, and use the acquisition image as a calibration image when the acquisition image meets the preset requirement, where the historical calibration image represents an image that meets the preset requirement acquired before the current acquisition moment.

[0130] In the embodiments of the present disclosure, an acquisition image obtained by the image acquisition device capturing a screen at the current acquisition moment is acquired, and then feature point detection is performed on the acquisition image, and the coverage information of the acquisition image is determined based on the detected feature points; the coverage information includes a corresponding area covered by the feature points on the field of view of the image acquisition device and / or a corresponding area covered by the feature points on the screen; according to the coverage information of the acquisition image and the coverage information of the historical calibration image, it is judged whether the acquisition image meets the preset requirement, and when the acquisition image meets the preset requirement, the acquisition image is used as a calibration image. In some scenarios, during the process of the acquisition personnel controlling the image acquisition device to capture the image on the screen or during the process of the image acquisition device capturing the image on the screen according to a preset rule, effective video frames can be screened out from the video stream in real time and automatically as the final calibration image; in other scenarios, after the acquisition personnel control the image acquisition device to capture the video stream and complete the data acquisition work, the images already acquired by the acquisition personnel can be judged, and effective video frames can be screened out as the final calibration image, realizing calibration image acquisition based on the video stream, avoiding redundant data such as low-quality, repeated, and non-contributing data to the area covered on the field of view of the image acquisition device and / or the area covered on the screen, ensuring the quality of the acquired calibration images; at the same time, the frequency of manual operations and the randomness and instability brought by manual acquisition are effectively reduced, improving the efficiency and stability of calibration image acquisition; in addition, compared with the existing method of using a calibration board to acquire calibration images, which requires all feature points to be covered in one acquisition, the method in the embodiments of the present disclosure does not require all feature points to be covered in one acquisition, and only needs the acquired image to meet the preset requirement, which is more convenient and efficient.

[0131] In a possible implementation, the determination module 903 is further configured to: determine the coverage rate at the current acquisition moment according to the coverage information of the acquired image and the coverage information of the historical calibration image, where the coverage rate represents the coverage degree of the acquired image and the historical calibration image with respect to the screen and / or the coverage degree with respect to the field of view of the image acquisition device; in the case where the coverage rate exceeds a first preset threshold, prompt the user to terminate the current acquisition work, and use the acquired image and the historical calibration image as the images for calibrating the image acquisition device.

[0132] In a possible implementation, the field of view of the image acquisition device includes a plurality of preset regions, and the corresponding region covered by the feature points on the field of view of the image acquisition device is represented by the preset region where the feature points are located; the preset requirement includes: at least one preset region where the feature points are located in the acquired image is different from all the preset regions where the feature points are located in the historical calibration image.

[0133] In a possible implementation, the screen includes a plurality of boxes, and the corresponding region covered by the feature points on the screen is represented by the box where the feature points are located; the preset requirement includes: at least one box where the feature points are located in the acquired image is different from all the boxes where the feature points are located in the historical calibration image.

[0134] In a possible implementation, the determination module 903 is further configured to: determine the acquisition moments corresponding to the respective images in the historical calibration image; screen out the target moments adjacent to the current acquisition moment from the acquisition moments corresponding to the respective images; the preset requirement further includes: the intersection-over-union ratio of all the boxes where the feature points are located in the acquired image and all the boxes where the feature points are located in the image corresponding to the target moment is less than a second preset threshold; or, the intersection-over-union ratio of all the preset regions where the feature points are located in the acquired image and all the preset regions where the feature points are located in the image corresponding to the target moment is less than a third preset threshold.

[0135] In a possible implementation, the determination module 903 is further configured to: divide the field of view of the image acquisition device into a plurality of preset regions; determine the coordinates of each feature point in the acquired image in the image coordinate system; and determine the preset region where each feature point in the acquired image is located according to the coordinates of each feature point in the acquired image in the image coordinate system.

[0136] In a possible implementation, the density of the feature points included in the image on the screen is determined by the focal length of the image acquisition device, where the density of the feature points is positively correlated with the focal length.

[0137] In a possible implementation, the determination module 903 is further configured to: determine the number of images in the historical calibration image; and in the case where the number of images is greater than a preset number, prompt the user to terminate the current acquisition work.

[0138] The above Figure 8 For the technical effects and specific descriptions of the calibration image acquisition device and its various possible implementations shown above, reference may be made to the above calibration image acquisition method, which will not be elaborated here.

[0139] It should be understood that the division of each module in the above device is only a logical function division. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. In addition, the modules in the device can be implemented in the form of a processor calling software; for example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or the functions of each module of the device, where the processor is, for example, a general-purpose processor, such as a Central Processing Unit (CPU) or a microprocessor, and the memory is a memory inside or outside the device. Or, the modules in the device can be implemented in the form of a hardware circuit, and the functions of some or all of the modules can be implemented through the design of the hardware circuit, and the hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of some or all of the above modules are implemented through the design of the logical relationship of the components in the circuit; again, in another implementation, the hardware circuit can be implemented by a programmable logic device (PLD). Taking a Field Programmable Gate Array (FPGA) as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured through a configuration file, so as to implement the functions of some or all of the above modules. All modules of the above device can be fully implemented in the form of a processor calling software, or fully implemented in the form of a hardware circuit, or partially implemented in the form of a processor calling software, and the remaining part is implemented in the form of a hardware circuit.

[0140] In an embodiment of the present disclosure, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a neural-network processing unit (NPU), a tensor processing unit (TPU), etc. In another implementation, the processor can implement certain functions through the logical relationship of hardware circuits, and the logical relationship of the hardware circuits is fixed or can be reconfigured. For example, the processor is a hardware circuit implemented by an ASIC or a PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the configuration of the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above modules.

[0141] It can be seen that each module in the above device can be one or more processors (or processing circuits) configured to implement the methods of the above embodiments. For example: CPU, GPU, NPU, TPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms. In addition, each module in the above device can be integrated in whole or in part, or can be independently implemented, and this is not limited.

[0142] The embodiment of the present disclosure also provides an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein, when the processor is configured to execute the instructions, the methods of the above embodiments are implemented. Exemplarily, the steps of the calibration image acquisition method shown in the above Figure 3 、 Figure 6 or Figure 7 can be executed.

[0143] Figure 9 FIG. shows a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. As Figure 9 shown, the electronic device may include: at least one processor 1001, a communication line 1002, a memory 1003, and at least one communication interface 1004.

[0144] The processor 1001 can be a general-purpose central processing unit, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits for controlling the execution of the programs of the present disclosure; the processor 1001 can also include a heterogeneous computing architecture of multiple general-purpose processors. For example, it can be a combination of at least two of CPU, GPU, microprocessor, DSP, ASIC, and FPGA; as an example, the processor 1001 can be CPU+GPU or CPU+ASIC or CPU+FPGA.

[0145] The communication line 1002 can include a path for transmitting information between the above components.

[0146] The communication interface 1004 uses any device of the transceiver type for communicating with other devices or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc.

[0147] The memory 1003 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or it can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory can exist independently and be connected to the processor through the communication line 1002. The memory can also be integrated with the processor. The memory provided by the embodiments of the present disclosure generally has non-volatility. Among them, the memory 1003 is used to store the computer execution instructions for executing the present disclosure solution, and is controlled by the processor 1001 to execute. The processor 1001 is used to execute the computer execution instructions stored in the memory 1003, thereby implementing the methods provided in the above embodiments of the present disclosure; exemplarily, it can implement the Figure 3 、 Figure 6 or Figure 7 steps of the calibration image acquisition method shown in.

[0148] Optionally, the computer-executable instructions in the embodiments of the present disclosure may also be referred to as application code, and the embodiments of the present disclosure do not make specific limitations thereto.

[0149] Exemplarily, the processor 1001 may include one or more CPUs. For example, Figure 9 the CPU0 in [[ ]]; the processor 1001 may also include one CPU and any one of GPU, ASIC, and FPGA. For example, Figure 9 the CPU0+GPU0 or CPU 0+ASIC0 or CPU0+FPGA0 in [[ ]].

[0150] Exemplarily, the electronic device may include multiple processors, such as Figure 9 the processor 1001 and the processor 1007 in [[ ]]. Each of these processors may be a single-core (single-CPU) processor, a multi-core (multi-CPU) processor, or a heterogeneous computing architecture including multiple general-purpose processors. Here, the processor may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0151] In a specific implementation, as an embodiment, the electronic device may further include an output device 1005 and an input device 1006. The output device 1005 communicates with the processor 1001 and can display information in various ways. For example, the output device 1005 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. For example, it may be a display device such as an in-vehicle HUD, an AR-HUD, or a monitor. The input device 1006 communicates with the processor 1001 and can receive user input in various ways. For example, the input device 1006 may be a mouse, a keyboard, a touch screen device, or a sensing device, etc.

[0152] The embodiments of the present disclosure provide a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the methods in the above embodiments are implemented. Exemplarily, the steps of the calibration image acquisition method shown in the above Figure 3 、 Figure 6 or Figure 7 can be implemented.

[0153] Embodiments of the present disclosure provide a computer program product, which may include, for example, computer-readable code or a non-volatile computer-readable storage medium carrying the computer-readable code; when the computer program product runs on a computer, the computer is caused to execute the methods in the above embodiments. Exemplarily, it may execute the steps of the calibration image acquisition method shown in the above Figure 3 , Figure 6 or Figure 7 .

[0154] The present disclosure may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0155] The computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punched card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium as used herein is not construed as an instantaneous signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0156] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or an external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. The network adapter or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0157] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present disclosure.

[0158] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer - readable program instructions.

[0159] These computer - readable program instructions can be provided to a processor of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data - processing apparatus, create a means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium that causes a computer, a programmable data - processing apparatus, and / or other devices to function in a particular manner, so that the computer - readable medium storing the instructions comprises a manufacture including instructions for implementing various aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0160] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other devices to generate a computer-implemented process, such that the instructions executed on the computer, other programmable data processing apparatus, or other devices implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0161] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of code, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0162] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A calibration image acquisition method, characterized in that, The method includes: By extracting video frames from the video stream captured in real time by the image acquisition device or the video stream that has been captured by the image acquisition device, an acquisition image obtained by the image acquisition device capturing the screen at the current acquisition moment is acquired, and an on-screen image containing feature points is displayed on the screen; Performing feature point detection on the acquisition image, and determining the coverage information of the acquisition image based on the detected feature points; the coverage information includes the corresponding area covered by the feature points on the field of view of the image acquisition device and / or the corresponding area covered by the feature points on the screen; According to the coverage information of the acquisition image and the coverage information of the historical calibration image, determining whether the acquisition image meets a preset requirement, and when the acquisition image meets the preset requirement, using the acquisition image as a calibration image, where the historical calibration image represents an image that meets the preset requirement acquired before the current acquisition moment.

2. The method according to claim 1, wherein After using the acquisition image as a calibration image, it further includes: Determining the coverage rate at the current acquisition moment according to the coverage information of the acquisition image and the coverage information of the historical calibration image, where the coverage rate represents the coverage degree of the acquisition image and the historical calibration image with respect to the screen and / or the coverage degree with respect to the field of view of the image acquisition device; Prompting the user to terminate the current acquisition work when the coverage rate exceeds a first preset threshold, and using the acquisition image and the historical calibration image as images for calibrating the image acquisition device.

3. The method according to claim 1, wherein The field of view of the image acquisition device includes multiple preset regions, and the corresponding region covered by the feature points on the field of view of the image acquisition device is represented by the preset region where the feature points are located; The preset requirement includes: At least one preset region where the feature points are located in the acquisition image is different from all the preset regions where the feature points are located in the historical calibration image.

4. The method according to claim 1, characterized in that, The screen includes multiple boxes, and the corresponding region covered by the feature points on the screen is represented by the box where the feature points are located; The preset requirement includes: At least one box where the feature points are located in the acquisition image is different from all the boxes where the feature points are located in the historical calibration image.

5. The method according to claim 3 or 4, characterized in that, The method further includes: Determining the acquisition moments corresponding to the respective images in the historical calibration image; Selecting target moments adjacent to the current acquisition moment from the acquisition moments corresponding to the respective images; The preset requirement further includes: The intersection over union of all the boxes where the feature points are located in the acquisition image and all the boxes where the feature points are located in the image corresponding to the target moment is less than a second preset threshold; Or, the intersection over union of all the preset regions where the feature points are located in the acquisition image and all the preset regions where the feature points are located in the image corresponding to the target moment is less than a third preset threshold.

6. The method according to claim 3, wherein The method further includes: Dividing the field of view of the image acquisition device into multiple preset regions; Determining the coordinates of each feature point in the acquisition image in the image coordinate system; According to the coordinates of each feature point in the acquisition image in the image coordinate system, determining the preset region where each feature point in the acquisition image is located.

7. The method according to claim 1, characterized in that, The density of feature points included in the on-screen image is determined by the focal length of the image acquisition device, where the density of feature points is positively correlated with the focal length.

8. The method according to claim 1, characterized in that, The method further includes: Determining the number of images in the historical calibration image; When the number of images is greater than a preset number, prompting the user to terminate the current acquisition work.

9. A calibration image acquisition device, characterized in that, The device includes: An acquisition module, configured to obtain an acquisition image obtained by the image acquisition device capturing a screen at the current acquisition moment by extracting video frames from a video stream captured in real time by the image acquisition device or a video stream that has been captured by the image acquisition device, where the screen displays an on-screen image including feature points; A detection module, configured to perform feature point detection on the acquisition image and determine coverage information of the acquisition image based on the detected feature points; the coverage information includes a corresponding area covered by the feature points on the field of view of the image acquisition device and / or a corresponding area covered by the feature points on the screen; A judgment module, configured to judge whether the acquisition image meets a preset requirement according to the coverage information of the acquisition image and the coverage information of the historical calibration image, and when the acquisition image meets the preset requirement, use the acquisition image as a calibration image, where the historical calibration image represents an image that meets the preset requirement acquired before the current acquisition moment.

10. An electronic device, characterized in that, Includes: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to implement the method according to any one of claims 1 to 8 when executing the instructions stored in the memory.

11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, The computer program instructions, when executed by the processor, implement the method according to any one of claims 1 to 8.

Images