Panoramic video inspection image display method

Abstract

The present invention provides a panoramic video inspection image display method, which constructs an effective visual range of an inspection route based on an inspection video of an inspected building object; spatially calibrates the inspection video walking route, namely, the SLAM inspection route, to obtain a spatial calibration result; based on the spatial calibration result, corrects the effective visual range of the SLAM inspection route to obtain an effective inspection range of the SLAM inspection route; based on the effective inspection range of the SLAM inspection route, marks the inspection blind area, can quickly and accurately calculate the image coverage area and the blind area, can accurately display the effective visual range of the panoramic image in a model or a plane drawing, and marks the inspection blind area, so as to better guide the inspection user to perform effective data collection.

Description

Technical Field

[0001] The invention relates to a panoramic video inspection image display method. Background Art

[0002] Panoramic video inspection based on SLAM has been widely used in the construction industry. By walking and shooting the construction site with panoramic video shooting equipment such as Yingshi, more comprehensive image data of the site can be obtained. At the same time, pure visual SLAM technology is used to solve the acquired video images to obtain the walking route of the inspection video, that is, the planar position of each video frame, which is of great help for on-site inspection management.

[0003] Currently, the more mature panoramic image processing and display platforms of solution providers can only display the positions of continuous route points after frame sampling on the map interface, and then switch to view the panoramic image by clicking on a certain route point. It is not clear how much effective visual range the panoramic image covers, and it is still impossible to determine which ranges can be clearly viewed through the image; at the same time, it is impossible to distinguish which areas cannot be clearly viewed and belong to the inspection blind spots. Summary of the invention

[0004] The purpose of the present invention is to provide a panoramic video inspection image display method.

[0005] In order to solve the above problems, the present invention provides a panoramic video inspection image display method, comprising:

[0006] Step 1: Based on the inspection video of the inspected building object, construct the effective visual range of the inspection route;

[0007] Step 2: Perform spatial calibration on the inspection video walking route, i.e., the SLAM inspection route, to obtain a spatial calibration result;

[0008] Step 3: Based on the spatial calibration result, the effective visual range of the SLAM inspection route is corrected to obtain the effective inspection range of the SLAM inspection route;

[0009] Step 4: Based on the effective inspection range of the SLAM inspection route, mark the inspection blind area.

[0010] Furthermore, in the above method, step 1: constructing an effective visual range of the inspection route based on the inspection video of the inspected building object includes:

[0011] Step 1.1: Obtain the inspection video after SLAM solution;

[0012] Step 1.2: Extract frames from the inspection video at equal time intervals to obtain a frame extraction result containing a continuous panoramic picture set;

[0013] Step 1.3: According to the frame extraction time of the inspection video, the position in the SLAM inspection route corresponding to the panoramic image of each frame in the frame extraction result is obtained, and the SLAM inspection route is composed of all the positions in the SLAM inspection route;

[0014] Step 1.4: Read the shooting parameters of each panoramic picture frame in the frame extraction result;

[0015] Step 1.5: Based on the set minimum image resolution, calculate the shooting distance that meets the minimum resolution requirement for each panoramic image under the current shooting parameters. ;

[0016] Step 1.6: Take the position of the SLAM inspection route corresponding to each frame of the panoramic image as the center of the circle, and calculate the value obtained in step 1.5. Draw a circle with the radius as the effective viewing range of each panoramic image frame;

[0017] Step 1.7: The effective visual range of all panoramic images on the current SLAM inspection route is merged to construct the effective visual range of the SLAM inspection route.

[0018] Furthermore, in the above method, the shooting parameters include: focal length, image pixel size and pixel number.

[0019] Furthermore, in the above method, the minimum resolution required photographic distance is , calculated by the following formula:

[0020] .

[0021] Furthermore, in the above method, step 2: spatially calibrating the inspection video walking route, i.e., the SLAM inspection route, to obtain a spatial calibration result, includes:

[0022] Step 2.1: Obtain the floor plan of the inspected building object according to the structural layer;

[0023] Step 2.2: Mark the starting point of the inspection on the floor plan according to the actual inspection walking position;

[0024] Step 2.3: Mark the end point of the inspection on the floor plan according to the actual inspection walking position;

[0025] Step 2.4: The SLAM inspection route calculated by SLAM is matched with the starting point and the ending point of the SLAM inspection route, respectively, with the starting point and the ending point marked in step 2.3 and step 2.4, so that the SLAM inspection route and the floor plan of the inspected building object are calibrated to complete the spatial geographic relationship.

[0026] Furthermore, in the above method, step 3: based on the spatial calibration result, correcting the effective visual range of the SLAM inspection route to obtain the effective inspection range of the SLAM inspection route includes:

[0027] Step 3.1: According to the frame extraction time of each inspection video in step 1.2, extract frames from the SLAM inspection route to obtain the plane position of each video frame corresponding to the plane layout diagram;

[0028] Step 3.2: According to the spatial calibration result of the inspection route in step 2, the effective visual range of each frame of the panoramic image in step 1 is superimposed on the plan layout of the inspected building object according to the corresponding plane position obtained in step 3.1 to obtain a first superimposed plan layout;

[0029] Step 3.3: Based on the first superimposed plan layout drawing, the positions of the wall and column type components of the inspected building object are marked to obtain a second superimposed plan layout drawing;

[0030] Step 3.4: Based on the second superimposed plan layout, for the area where the effective visible range of each panoramic image intersects and collides with the wall and column type components, in the fan-shaped area from the camera position of the panoramic image to the colliding wall and column type components, the side of the colliding wall and column type components close to the camera position is marked as the visible range, and the area on the other side of the colliding wall and column type components is marked as the blind area, so as to obtain a third superimposed plan layout;

[0031] Step 3.5: According to the area marked as the visible range in the third superimposed plan layout diagram, on the basis of the third superimposed plan layout diagram, reconstruct the boundary line of the effective visible range of the SLAM inspection route, and obtain the effective inspection range of the SLAM inspection route based on the boundary line to obtain the fourth superimposed plan layout diagram.

[0032] Furthermore, in the above method, step 4: marking the inspection blind area based on the effective inspection range of the SLAM inspection route includes:

[0033] Based on the effective inspection range of the SLAM inspection route in the fourth superimposed plan layout, the remaining area outside the effective inspection range of the SLAM inspection route is marked as an inspection blind area on the basis of the fourth superimposed plan layout to obtain a fifth superimposed plan layout.

[0034] Furthermore, in the above method, after step 4, step 5 is also included:

[0035] Acquire the user's click operation within the effective inspection range of the SLAM inspection route in the fifth superimposed plan view, calculate the position in the SLAM inspection route closest to the plane position corresponding to the click operation based on the plane position corresponding to the click operation, display a frame of panoramic picture corresponding to the position in the nearest SLAM inspection route, and automatically switch the panoramic perspective of the corresponding frame of panoramic picture to the direction of the plane position corresponding to the click operation.

[0036] According to another aspect of the present invention, there is also provided a computer device, comprising:

[0037] Processor; and

[0038] A memory arranged to store computer executable instructions which, when executed, cause the processor to: adopt a method as described in any one of the above.

[0039] According to another aspect of the present invention, there is further provided a computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by a processor, the processor is caused to: adopt any of the methods described above.

[0040] Compared with the prior art, in order to solve the needs of marking the effective inspection range and inspection blind spots of the above-mentioned panoramic video image inspection, the present invention constructs an effective visual range of the inspection route based on the inspection video of the inspected building object; spatially calibrates the inspection video walking route, i.e., the SLAM inspection route, to obtain a spatial calibration result; based on the spatial calibration result, the effective visual range of the SLAM inspection route is corrected to obtain the effective inspection range of the SLAM inspection route; based on the effective inspection range of the SLAM inspection route, the inspection blind spot is marked, and the image coverage area and blind spot area can be quickly and accurately calculated, and the effective visual range of the panoramic image can be accurately displayed in the model or plane drawing, and the inspection blind spot can be marked, so as to better guide the inspection users to collect data effectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the effective visual range of the inspection route of an embodiment of the present invention;

[0042] Figure 2 It is a schematic diagram of superimposing the effective visual range of the inspection route and the plan layout diagram according to an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of optimizing the effective visual range of inspection according to an embodiment of the present invention;

[0044] Figure 4 It is a schematic diagram of the inspection range of an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0046] In a typical configuration of the present application, the terminal, the device of the service network and the trusted party all include one or more processors (CPU), input / output interfaces, network interfaces and memories.

[0047] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0048] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0049] The present invention provides a panoramic video inspection image display method, the method comprising:

[0050] Step 1: Based on the inspection video of the inspected building object, construct the effective visual range of the inspection route;

[0051] Step 1.1: Obtain the inspection video after SLAM solution;

[0052] Step 1.2: Extract frames from the inspection video at equal time intervals to obtain a frame extraction result containing a continuous panoramic picture set;

[0053] Step 1.3: According to the frame extraction time of the inspection video, the position in the SLAM inspection route corresponding to the panoramic image of each frame in the frame extraction result is obtained, and the SLAM inspection route is composed of all the positions in the SLAM inspection route;

[0054] Step 1.4: Read the shooting parameters of each panoramic image frame in the frame extraction result, such as focal length, image pixel size, number of pixels, etc. (panoramic cameras generally have autofocus and zoom lenses);

[0055] Step 1.5: Based on the set minimum image resolution, calculate the shooting distance that can meet the minimum resolution requirement for each panoramic image under the current shooting parameters. ;

[0056] Step 1.6: Take the position of the SLAM inspection route corresponding to each frame of the panoramic image as the center of the circle, and calculate the value obtained in step 1.5. D Draw a circle with the radius as the effective viewing range of each panoramic image frame;

[0057] Step 1.7: Merge the effective visual range of all panoramic images on the current SLAM inspection route to construct the effective visual range of the SLAM inspection route. Figure 1 .

[0058] Step 2: Perform spatial calibration on the inspection video walking route, i.e., the SLAM inspection route, to obtain a spatial calibration result;

[0059] Step 2.1: Obtain the floor plan of the inspected building object according to the structural layer;

[0060] Step 2.2: Mark the starting point of the inspection on the floor plan according to the actual inspection walking position;

[0061] Step 2.3: Mark the end point of the inspection on the floor plan according to the actual inspection walking position;

[0062] Step 2.4: The SLAM inspection route calculated by SLAM is matched with the starting point and the ending point of the SLAM inspection route, respectively, with the starting point and the ending point marked in step 2.3 and step 2.4, so that the SLAM inspection route and the floor plan of the inspected building object are calibrated to complete the spatial geographic relationship.

[0063] Step 3: Based on the spatial calibration result, the effective visual range of the SLAM inspection route is corrected to obtain the effective inspection range of the SLAM inspection route;

[0064] Step 3.1: According to the time of extracting frames of each inspection video in step 1.2, extract frames from the SLAM inspection route to obtain the plane position of each video frame corresponding to the plane layout diagram;

[0065] Step 3.2: Based on the spatial calibration results of the inspection route in step 2, the effective visual range of each frame of the panoramic image in step 1 is superimposed on the floor plan of the inspected building object according to the corresponding plane position obtained in step 3.1, see the attached Figure 2 , to obtain a first superimposed plan layout;

[0066] Step 3.3: Based on the first superimposed plan layout, mark the locations of the wall and column type components of the inspected building object, see the attached Figure 2 , to obtain a second superimposed plan layout;

[0067] Step 3.4: Based on the second superimposed plan layout, for the area where the effective visible range of each panoramic image intersects and collides with the wall and column type components, in the fan-shaped area from the camera position of the panoramic image to the colliding wall and column type components, the side of the colliding wall and column type components close to the camera position is marked as the visible range, and the area on the other side of the colliding wall and column type components is marked as the blind area, see Attachment 2. Figure 3 , to obtain a third superimposed plan layout;

[0068] Step 3.5: According to the area marked as the visible range in the third superimposed plan layout, on the basis of the third superimposed plan layout, the boundary line of the effective visible range of the SLAM inspection route is reconstructed, and the effective inspection range of the SLAM inspection route is obtained based on the boundary line, see Attachment 3. Figure 4 , to obtain a fourth superimposed plan layout;

[0069] Step 4: Based on the effective inspection range of the SLAM inspection route in the fourth superimposed plan layout, the remaining area outside the effective inspection range of the SLAM inspection route is marked as an inspection blind area on the basis of the fourth superimposed plan layout to obtain a fifth superimposed plan layout;

[0070] Here, according to the effective inspection range of the SLAM inspection route obtained in step 3, the remaining area can be indicated as an inspection blind area in the floor plan.

[0071] Preferably, after step 4, step 5 may also be included:

[0072] Acquire the user's click operation within the effective inspection range of the SLAM inspection route in the fifth superimposed plan view, calculate the position in the SLAM inspection route closest to the plane position corresponding to the click operation based on the plane position corresponding to the click operation, display a frame of panoramic picture corresponding to the position in the nearest SLAM inspection route, and automatically switch the panoramic perspective of the corresponding frame of panoramic picture to the direction of the plane position corresponding to the click operation.

[0073] To summarize, in order to solve the above-mentioned needs of panoramic video image inspection to mark the effective inspection range and inspection blind spots, the present invention constructs an effective visual range of the inspection route based on the inspection video of the inspected building object; spatially calibrates the inspection video walking route, i.e., the SLAM inspection route, to obtain a spatial calibration result; based on the spatial calibration result, corrects the effective visual range of the SLAM inspection route to obtain the effective inspection range of the SLAM inspection route; based on the effective inspection range of the SLAM inspection route, marks the inspection blind spots, can quickly and accurately calculate the image coverage area and blind spots, can accurately display the effective visual range of the panoramic image in the model or plane drawing, and mark the inspection blind spots, so as to better guide the inspection users to collect effective data.

[0074] According to another aspect of the present invention, there is also provided a computer device, comprising:

[0075] Processor; and

[0076] A memory arranged to store computer executable instructions, which, when executed, cause the processor to: adopt a method as described in any of the above embodiments.

[0077] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein when the computer-executable instructions are executed by a processor, the processor is caused to: adopt a method as described in any of the above embodiments.

[0078] The detailed contents of the device embodiments of the present invention can be found in the corresponding parts of the method embodiments, which will not be described in detail here.

[0079] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0080] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present invention (including related data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present invention can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.

[0081] In addition, a part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. The program instruction for calling the method of the present invention may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of a computer device that runs according to the program instruction. Here, according to an embodiment of the present invention, a device is included, the device including a memory for storing computer program instructions and a processor for executing program instructions, wherein, when the computer program instruction is executed by the processor, the device is triggered to run the method and / or technical solution based on the aforementioned multiple embodiments of the present invention.

[0082] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A panoramic video inspection image display method, characterized in that: include: Step 1: Based on the inspection video of the inspected building object, construct the effective visual range of the inspection route; Step 2: Perform spatial calibration on the inspection video walking route, i.e., the SLAM inspection route, to obtain a spatial calibration result; Step 3: Based on the spatial calibration result, the effective visual range of the SLAM inspection route is corrected to obtain the effective inspection range of the SLAM inspection route; Step 4: Based on the effective inspection range of the SLAM inspection route, mark the inspection blind area; Step 1: Based on the inspection video of the inspected building object, construct the effective visual range of the inspection route, including: Step 1.1: Obtain the inspection video after SLAM solution; Step 1.2: Extract frames from the inspection video at equal time intervals to obtain a frame extraction result containing a continuous panoramic picture set; Step 1.3: According to the frame extraction time of the inspection video, the position in the SLAM inspection route corresponding to the panoramic image of each frame in the frame extraction result is obtained, and the SLAM inspection route is composed of all the positions in the SLAM inspection route; Step 1.4: Read the shooting parameters of each panoramic picture frame in the frame extraction result; Step 1.5: According to the set minimum image resolution, calculate the shooting distance D that meets the minimum resolution requirement for each panoramic image frame under the current shooting parameters; Step 1.6: Take the position in the SLAM inspection route corresponding to each frame of the panoramic image as the center of the circle and the D calculated in step 1.5 as the radius to draw a circle, which is the effective visual range of each frame of the panoramic image; Step 1.7: Merge the effective visual ranges of all panoramic images on the current SLAM inspection route to construct the effective visual range of the SLAM inspection route; The shooting parameters include: focal length, image pixel size and pixel number; The photography distance D required for the minimum resolution is calculated by the following formula: Step 2: Perform spatial calibration on the inspection video walking route, i.e., the SLAM inspection route, to obtain the spatial calibration result, including: Step 2.1: Obtain the floor plan of the inspected building object according to the structural layer; Step 2.2: Mark the starting point of the inspection on the floor plan according to the actual inspection walking position; Step 2.3: Mark the end point of the inspection on the floor plan according to the actual inspection walking position; Step 2.4: Match the SLAM inspection route calculated by SLAM, the starting point and the ending point of the SLAM inspection route, with the starting point and the ending point marked in step 2.3 and step 2.4, respectively, so that the SLAM inspection route and the floor plan of the inspected building object are calibrated to complete the spatial geographic relationship; Step 3: Based on the spatial calibration result, the effective visual range of the SLAM inspection route is corrected to obtain the effective inspection range of the SLAM inspection route, including: Step 3.1: According to the frame extraction time of each inspection video in step 1.2, extract frames from the SLAM inspection route to obtain the plane position of each video frame corresponding to the plane layout diagram; Step 3.2: According to the spatial calibration result of the inspection route in step 2, the effective visual range of each frame of the panoramic image in step 1 is superimposed on the plan layout of the inspected building object according to the corresponding plane position obtained in step 3.1 to obtain a first superimposed plan layout; Step 3.3: Based on the first superimposed plan layout drawing, the positions of the wall and column type components of the inspected building object are marked to obtain a second superimposed plan layout drawing; Step 3.4: Based on the second superimposed plan layout, for the area where the effective visible range of each panoramic image intersects and collides with the wall and column type components, in the fan-shaped area from the camera position of the panoramic image to the colliding wall and column type components, the side of the colliding wall and column type components close to the camera position is marked as the visible range, and the area on the other side of the colliding wall and column type components is marked as the blind area, so as to obtain a third superimposed plan layout; Step 3.5: According to the area marked as the visible range in the third superimposed plan layout diagram, on the basis of the third superimposed plan layout diagram, reconstruct the boundary line of the effective visible range of the SLAM inspection route, and obtain the effective inspection range of the SLAM inspection route based on the boundary line to obtain the fourth superimposed plan layout diagram.

2. The panoramic video inspection image display method according to claim 1, characterized in that: Step 4: Based on the effective inspection range of the SLAM inspection route, mark the inspection blind areas, including: Based on the effective inspection range of the SLAM inspection route in the fourth superimposed plan layout, the remaining area outside the effective inspection range of the SLAM inspection route is marked as an inspection blind area on the basis of the fourth superimposed plan layout to obtain a fifth superimposed plan layout.

3. The panoramic video inspection image display method according to claim 2, characterized in that: After step 4, step 5 is also included: Acquire the user's click operation within the effective inspection range of the SLAM inspection route in the fifth superimposed plan view, calculate the position in the SLAM inspection route closest to the plane position corresponding to the click operation based on the plane position corresponding to the click operation, display a frame of panoramic picture corresponding to the position in the nearest SLAM inspection route, and automatically switch the panoramic perspective of the corresponding frame of panoramic picture to the direction of the plane position corresponding to the click operation.

4. A computer-readable storage medium having computer-executable instructions stored thereon, wherein: When the computer executable instructions are executed by a processor, the processor is caused to: adopt the method according to any one of claims 1 to 3.

5. A computer device, wherein: include: processor; as well as A memory arranged to store computer executable instructions which, when executed, cause the processor to: employ a method as claimed in any one of claims 1 to 3.

Images