Inspection system, inspection method, and program

Abstract

To improve the efficiency of inspection work.SOLUTION: An inspection system includes a photographing unit that acquires an image of a real object, a leveling rod generation unit that generates one or more leveling rod images to be compared with the object, and a display control unit that superimposes and displays the leveling rod image on the image, wherein the display control unit determines an arrangement of the leveling rod image based on a dimension line included in a drawing of the object, adjusts a scale of the leveling rod image to be equal to a scale of the real object, and superimposes and displays the leveling rod image on the image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an inspection system, an inspection method, and a program. [Background technology]

[0002] Material inspection (hereinafter referred to as inspection) is the process of checking whether materials have been delivered according to the specified specifications. During inspection, we confirm that the shape and dimensions of the object are as designed, and as proof of this, we place a measuring rod on the object and take a photograph.

[0003] Although Patent Document 1 is not related to material inspection, it describes a system for recording that reinforcement has been arranged as designed at a construction site. This system displays a staff superimposed on a construction image taken at the site. The user can specify the display position of the staff as desired. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-010395 Summary of the Invention [Problem to be solved by the invention]

[0005] In acceptance inspection, the more points that need to be certified, the more staffs are required, which requires more time and effort. In the method described in Patent Document 1, the user must specify the display position of the staff. Therefore, even if the method described in Patent Document 1 is simply applied to acceptance inspection, the more points that need to be certified, the more time it takes to specify the display position of the staff. Therefore, improving the efficiency of acceptance inspection work becomes an issue. [Means for solving the problem]

[0006] According to one embodiment, the inspection system includes a photographing unit that captures an image of a real object, a staff generation unit that generates one or more staff images for comparison with the object, and a display control unit that superimposes the staff image on the image, wherein the display control unit determines the position of the staff image based on dimension lines included in a drawing of the object, adjusts the scale of the staff image so that it is the same size as the real object, and superimposes the staff image on the image. According to one embodiment, the inspection system further includes an electronic whiteboard generation unit that generates an electronic whiteboard image containing information necessary for on-site photographs, the electronic whiteboard including the drawing of the object, and the display control unit determines the placement of the scale image based on the dimension lines included in the drawing included on the electronic whiteboard. According to one embodiment, the inspection method is a computer-implemented method that includes a photographing step for capturing an image of a real object, a staff generation step for generating one or more staff images for comparison with the object, and a display control step for superimposing the staff images on the image, wherein the display control step includes a step for determining the position of the staff image based on dimension lines included in a drawing of the object, a step for adjusting the scale of the staff image so that it is the same size as the real object, and a step for superimposing the staff image on the image. According to one embodiment, a program causes a computer to carry out the above method. [Effects of the Invention]

[0007] The present invention provides an inspection system, an inspection method, and a program that can improve the efficiency of inspection work. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of a hardware configuration of an inspection system 1. FIG. [Figure 2] 2 is a block diagram showing an example of a functional configuration of the inspection system 1. FIG. [Figure 3] FIG. 10 is a diagram showing an example of a processing drawing. [Figure 4] FIG. 10 is a diagram showing an example of an image of an electronic whiteboard. [Figure 5] FIG. 10 is a diagram showing an example of a staff image. [Figure 6] FIG. 10 is a diagram showing an example of the arrangement of a staff image. [Figure 7] FIG. 10 is a diagram showing an example of the arrangement of a staff image. [Figure 8] FIG. 10 is a diagram showing an example of the arrangement of a staff image. [Figure 9] FIG. 10 is a diagram showing an example of a procedure for adjusting the scale of a staff image. [Figure 10] FIG. 10 is a diagram showing an example of a superimposed display of a staff image. [Figure 11] FIG. 10 is a diagram showing an example of a site photograph on which an electronic whiteboard image and a staff image are superimposed. [Figure 12] 4 is a flowchart showing an example of the operation of the inspection system 1. [Figure 13] FIG. 10 is a diagram showing an example of acceptance inspection using an actual staff. [Figure 14] FIG. 10 is a diagram showing an example of acceptance inspection using an actual staff. [Figure 15] FIG. 10 is a diagram showing an example of acceptance inspection using an actual staff. [Figure 16] FIG. 10 is a diagram showing an example of acceptance inspection using an actual staff. [Figure 17] FIG. 10 is a diagram showing an example of acceptance inspection using an actual staff. [Figure 18] FIG. 10 is a diagram showing an example of acceptance inspection using an actual staff. DETAILED DESCRIPTION OF THE INVENTION

[0009] Below are some examples of cases where inspections are carried out at construction sites, etc. When inspecting processed rebars to be used in concrete structures, it is confirmed that the diameter and shape dimensions of the rebars are as designed, and as proof of this, a staff is placed on each processed part and photographs are taken (see Figure 13). When inspecting catch basins (devices for draining rainwater from the bridge surface) used in bridge decks, it is confirmed that the diameter and shape dimensions of the openings are as designed, and as proof of this, a scale is placed along the shape of the catch basin and a photograph is taken (see Figure 14). When inspecting the bearings used in the main girders of a bridge (devices that are installed between the superstructure and substructure of the bridge to support the girders), it is confirmed that the diameter, spacing, and shape dimensions of the rebars are as designed, and as proof of this, a scale is placed along the shape of the bearing and a photograph is taken (see Figure 15). When inspecting the expansion joints of the bridge's main girders (the devices that allow clearance between abutments and girders to accommodate expansion and contraction of the bridge girders), it is confirmed that the shape and dimensions of the components are as designed, and as proof of this, a scale is placed along the shape of the expansion joint and a photograph is taken (see Figure 16). When inspecting the concrete girders that make up the main girders of a bridge, it is confirmed that the shape and dimensions of the components are as designed, and as proof of this, a scale is placed along the shape of the components and photographs are taken (see Figures 17 and 18).

[0010] The present invention can be applied to the inspection of all of the above-mentioned objects, but is not limited to these cases. It can be applied to the inspection of any object that is manufactured in a factory or the like and delivered to a construction site or the like, and where it is expected that a staff will be placed on the object and on-site photographs will be taken. In the following embodiments, for the sake of convenience, the inspection of processed rebar will be mainly described as an example.

[0011] (Embodiment 1) 1 is a diagram illustrating an example of a hardware configuration of an acceptance inspection system 1. The acceptance inspection system 1 includes a first terminal device 10, a second terminal device 20, and a server 30.

[0012] The first terminal device 10 is an information processing device including a processor, a memory, a display for presenting information to a user, an input device (touch panel, eye-gaze input device, pointing device, etc.) for accepting information input from a user, a communication device, a camera for capturing images, etc. It may further include various sensors (gyro, acceleration sensor, depth sensor, ToF sensor, etc.). The first terminal device 10 is typically a tablet computer, a smartphone, an HMD (e.g., HoloLens, etc.), etc. The first terminal device 10 is typically used by a user who performs acceptance inspection at a construction site, etc.

[0013] The second terminal device 20 is an information processing device including a processor, a memory, a display for presenting information to a user, an input device (keyboard, pointing device, etc.) for receiving information input from a user, a communication device, etc. The second terminal device 20 is typically a PC (Personal Computer). The second terminal device 20 is typically used by a user who manages or assists in inspection in an office.

[0014] The server 30 is an information processing device including a processor, a memory, a communication device, etc. The server 30 is typically a server computer. Note that the server 30 does not necessarily have to be a single computer, and may be realized by multiple computers. For example, the server 30 may be realized in a cloud computing environment. The server 30 is connected to the first terminal device 10 and the second terminal device 20 so as to be able to communicate with each other.

[0015] 2 is a diagram showing an example of the functional configuration of the inspection system 1. The inspection system 1 includes a photographing unit 101, an electronic whiteboard generation unit 102, a staff generation unit 103, a display control unit 104, an electronic whiteboard information input unit 201, and a storage unit 301.

[0016] In this embodiment, it is assumed that each functional unit is realized by the following hardware: The first terminal device 10 has a photographing unit 101, an electronic whiteboard generating unit 102, a staff generating unit 103, and a display control unit 104. The second terminal device 20 has an electronic whiteboard information input unit 201. The server 30 has a storage unit 301.

[0017] Note that this is merely one implementation example, and the present invention is not limited to this example. For example, both the first terminal device 10 and the second terminal device 20 may be realized by the same device (for example, one tablet computer). In this case, the photographing unit 101, the electronic whiteboard generating unit 102, the staff generating unit 103, the display control unit 104, and the electronic whiteboard information input unit 201 are implemented in one device. Furthermore, the server 30 may be the same device as the first terminal device 10 or the second terminal device 20. In this case, the storage unit 301 is implemented in the first terminal device 10 or the second terminal device 20.

[0018] (Camera Section 101) At construction sites, etc., it is common practice to take photographs showing the site conditions (hereinafter referred to as site photographs) and store them to record the construction results. In this embodiment, this is achieved electronically mainly by the photographing unit 101.

[0019] A typical use case is assumed in which a user inspecting an object to be inspected, typically processed rebar, uses the photographing unit 101 of a tablet computer, which is the first terminal device 10, to photograph the object.

[0020] In response to a user instruction (starting the electronic viewfinder mode), the photographing unit 101 controls the camera to photograph an image of the site including the inspection object. The image is displayed on the display in approximately real time. In response to a user instruction (pressing the shutter button), the photographing unit 101 generates image data (site photo) including the camera image at that time.

[0021] As will be described later, an electronic whiteboard image and a staff image can be superimposed on the video of the site (display control unit 104). When the user presses the shutter button in this state, the photographing unit 101 creates a site photograph that captures the inspection object, the electronic whiteboard image, and the staff image. The photographing unit 101 stores the created site photograph as image data in, for example, the storage unit 301 of the server 30.

[0022] (Electronic whiteboard generation unit 102) Conventionally, when taking on-site photographs, information required for inspection would be written on a blackboard with chalk or the like and then copied onto the on-site photograph. In recent years, electronic blackboard systems, which are electronic versions of such blackboards, have also come into use. The electronic blackboard generation unit 102 cooperates with the electronic blackboard information input unit 201 (described later) to realize the electronic blackboard on the inspection system 1.

[0023] It is assumed that the items to be displayed on the electronic whiteboard are stored in advance in the storage unit 301 of the server 30 by the electronic whiteboard information input unit 201 described below. The electronic whiteboard generation unit 102 references the storage unit 301 of the server 30 and acquires the items to be displayed on the electronic whiteboard. Taking processed rebar as an example, the items to be displayed on the electronic whiteboard include text information such as the name of the project, the type of work, the rebar diameter, and the rebar number (symbol). A shape diagram of the rebar may also be included. As the shape diagram, for example, a CAD drawing file of a processing drawing is stored in the storage unit 301.

[0024] An example of a processing drawing is shown in Figure 3. This CAD drawing shows the shape of the rebar body, as well as dimension lines showing the dimensions of each part. The electronic whiteboard generation unit 102 can convert the CAD drawing into image data, for example, and include the image data of the processing drawing in the electronic whiteboard image.

[0025] The electronic whiteboard generation unit 102 formats the acquired information into a predetermined format and generates an electronic whiteboard image. An example of an electronic whiteboard image is shown in Figure 4. Note that the format of the electronic whiteboard image is not limited to the example in Figure 4, and any table format, color, font, layout, etc. can be used.

[0026] (Level generation unit 103) The staff generation unit 103 generates an image of a staff (hereinafter referred to as a staff image). A staff is a scale that is captured in a site photograph together with an object for comparison with the object. Conventionally, a rod-shaped scale (actual object) was placed next to processed rebars or the like at the site, and site photographs were taken. In this embodiment, an electronically generated staff image replaces the conventional actual scale.

[0027] An example of a staff image is shown in Figure 5. The staff image is a strip-shaped image with dimensional graduations engraved on it, and is an imitation of an actual scale. It is preferable that the staff generation unit 103 generates a staff image of a predetermined length each time in response to a request from the display control unit 104 (described later). It is also possible to store staff images of several lengths in a predetermined storage area in advance, and read out a staff image of a predetermined length in response to a request from the display control unit 104.

[0028] (Display control unit 104) The display control unit 104 superimposes (composites) the electronic whiteboard image generated by the electronic whiteboard generation unit 102 and the image generated by the staff generation unit 103 on the video generated by the imaging unit 101. The superimposition can be performed, for example, in response to a user instruction (a superimposition start instruction). That is, when a video of the site is displayed on the display in approximately real time and the first terminal device 10 is functioning as an electronic viewfinder, and the user issues a superimposition start instruction by operating a button or the like, the display control unit 104 displays the electronic whiteboard image and the staff image superimposed on the video of the site. Alternatively, the display control unit 104 may always superimpose the electronic whiteboard image and the staff image on the video of the site while the video of the site is displayed on the display in approximately real time and the first terminal device 10 is functioning as an electronic viewfinder. Alternatively, the display control unit 104 may have a function for recognizing objects in the video and, when it estimates that an inspection target is included in the video of the site, start superimposing the electronic whiteboard image and the staff image on the video of the site. In either case, the electronic whiteboard image and the staff image may be superimposed and displayed in an opaque state, or may be superimposed and displayed with a predetermined transparency to make it easier to see the situation on site (items to be inspected, etc.).

[0029] The electronic whiteboard image is placed at a predetermined position in the video of the site. For example, it may be placed near one of the four corners of the field of view, where it is unlikely to interfere with the staff. It is preferable that the display control unit 104 moves and enlarges / reduces the electronic whiteboard image in response to user operations (dragging, pinching in / out, etc.).

[0030] The display control unit 104 can automatically determine the arrangement of one or more staff images (number, length, position, etc. of the staffs) according to the shape of the inspection object. For example, if the inspection object is a processed rebar, as shown in Fig. 6, the staff image is arranged so that one end point of the staff is aligned with one end point of the rebar. This allows the user to recognize the length of the rebar by referring to the staff image.

[0031] In addition, as shown in Figure 7, one end of the staff may be aligned with the position of the processed part (bending) of the processed rebar (the start or peak position of the bending). This allows the length from the end to the processed part, or from the processed part to the processed part, to be measured.

[0032] Also, when multiple straight sections of a processed rebar face each other, as shown in Figure 8, the staff may be positioned so that it is placed from one end point of the rebar toward the opposing straight section. This allows the distance between the straight sections of the processed rebar to be measured. The display control unit 104 may draw auxiliary lines at any point that can be obtained from the processing drawing, such as the processed section or end point, as shown by the dashed line in Figure 8.

[0033] To realize such various arrangements, the display control unit 104 can determine the arrangement form of the staff images (the number, length, position, etc. of the staffs) using the dimension lines included in the processing drawing acquired by the electronic whiteboard generation unit 102. First, the display control unit 104 uses the coordinates of the start point and end point of one dimension line included in the processing drawing (coordinates in the model coordinate system of the processing drawing) as the coordinates of the viewpoint and end point of the staff, and causes the staff generation unit 103 to generate one or more staff images (expressed in the model coordinate system of the processing drawing). Next, the display control unit 104 enlarges / reduces the staff image so that it is the same size in real space according to the scale of the processing drawing. Thereafter, the scale-adjusted staff image is projected into real space. Here, projection into real space means superimposing and displaying the scale-adjusted staff image on the captured image of real space displayed on the display.

[0034] An example of a procedure for adjusting the scale of a staff image is shown below. The display control unit 104 performs spatial recognition using the camera and various sensors (such as a gyro, acceleration sensor, depth sensor, and ToF sensor) provided in the first terminal device 10. This enables scale conversion between the model coordinate system and the coordinate system of real space. Note that slight scale errors may occur depending on the on-site environment, which may result in an error in the size of the projected staff. To mitigate this problem, two AR markers are placed at a predetermined distance (e.g., 1,000 mm) and the first terminal device 10 is made to recognize these two points (FIG. 9). This allows for more accurate measurement of the real space and placement of the staff at the correct scale. Alternatively, calibration can be performed by having the first terminal device 10 recognize two points with a known distance that are fixed in the environment, without placing two AR markers each time.

[0035] The staff image, scaled to the same size as the real space, is placed at a predetermined position in the real space (FIG. 10). Typically, the endpoint of the predetermined staff can be aligned with the installation position of the AR marker. The display control unit 104 then moves and rotates the staff image as desired in response to user operations (drag, rotation, etc.). This allows the user to move the staff image to a position that matches the processed rebar placed in the real space, adjust the angle, and create a state in which the dimensions of each part of the rebar can be confirmed on the image.

[0036] By issuing an image generation command (pressing the shutter button) in this state, the user can generate an on-site photo of the processed rebar with the staff image and electronic whiteboard image superimposed (Fig. 11). This eliminates the need to manually set up the actual staff each time, making inspection work more efficient.

[0037] (Electronic whiteboard information input unit 201) The electronic whiteboard information input unit 201 accepts input of items to be displayed on the electronic whiteboard and stores them in the memory unit 301 of the server 30. The information stored in the memory unit 301 is acquired by the electronic whiteboard generation unit 102 and used to generate an electronic whiteboard image.

[0038] A typical use case is assumed to be when a user who manages or assists with inspection inputs information using a PC, which is the second terminal device 20, in an office or the like.

[0039] The electronic whiteboard information input unit 201 provides an interface for inputting information to be displayed on the electronic whiteboard. Taking processed rebar as an example, the information to be displayed on the electronic whiteboard includes text information such as the name of the project, the type of work, the rebar diameter, and the rebar number (symbol). The electronic whiteboard information input unit 201 displays text boxes, etc. for inputting this information on the display and accepts text input via a keyboard, etc. The electronic whiteboard information input unit 201 also accepts input of a shape drawing of the object to be inspected. Taking processed rebar as an example, the electronic whiteboard information input unit 201 allows the user to select, for example, a CAD drawing file of the processing drawing. The electronic whiteboard information input unit 201 stores the input text information, files, etc. in the storage unit 301.

[0040] (Storage unit 301) The storage unit 301 of the server 30 receives and stores various information to be displayed on the electronic whiteboard from the electronic whiteboard information input unit 201. It can also function as a storage for storing on-site photographs generated by the photographing unit 101.

[0041] An example of the operation of the inspection system 1 will be described with reference to the flowchart of FIG.

[0042] S1: Input the information to be displayed on the electronic whiteboard A user who manages or assists with inspections uses the second terminal device 20 in an office or the like to input information to be displayed on the electronic whiteboard. The electronic whiteboard information input unit 201 accepts input of various text information for identifying the construction work or the object to be inspected, as well as a shape diagram (CAD drawing) of the object to be inspected.

[0043] S2: Shooting on-site footage A user who is to carry out an inspection takes an image of the inspection object using the first terminal device 10. The image capturing unit 101 captures an image of the site including the inspection object, and displays the image on a display.

[0044] S3: Generation of electronic whiteboard images The interactive whiteboard generation unit 102 generates an image of the interactive whiteboard using the information input in S1.

[0045] S4: Creation of staff image The staff generation unit 103 generates a staff image.

[0046] S5: Superimposed display of electronic whiteboard image and staff image The display control unit 104 superimposes the electronic whiteboard image generated in S3 and the staff image generated in S4 on the video of the site captured in S2. At this time, the display control unit 104 can automatically determine the arrangement of the staff images (number, length, position, etc. of the staffs) according to the shape of the object to be inspected. Specifically, the staff images are arranged based on the dimension lines included in the shape drawing of the object acquired in S1.

[0047] S6: Generate site photos For example, when a user presses a shutter button, the photographing unit 101 generates a site photograph, i.e., image data, including the video of the site displayed on the display at that time, an image of the electronic whiteboard, and an image of a staff. The photographing unit 101 stores the generated site photograph in, for example, the storage unit 301.

[0048] In the first embodiment, the display control unit 104 generates a site photograph in which an electronic whiteboard image and a staff image are superimposed on a video of the site. At this time, the display control unit 104 automatically determines the arrangement of the staff images (number, length, position, etc. of the staffs) according to the shape of the inspection object. Even if the inspection object includes many processed parts, the required number of staffs can be displayed at appropriate lengths and positions without requiring the user's effort. This improves the efficiency of the inspection work.

[0049] (Embodiment 2) In the first embodiment, a staff image is superimposed on a site photograph. In the second embodiment, an actual staff is placed at the site, and both the staff image and the actual staff are captured in the site photograph. This allows an observer of the site photograph to compare the actual staff captured in the site photograph with the electronically generated staff image and verify the accuracy of the staff image.

[0050] (Embodiment 3) Instead of generating a staff image each time according to the dimension line, one or more staff image arrangement patterns can be created and saved in advance, and can be recalled and used as needed. For example, if arrangement patterns corresponding to typical rebar processing shapes such as U-shapes and U-shaped ones are created in advance, the appropriate arrangement pattern can be retrieved and used without using a processing drawing.

[0051] (Other embodiments) In the above-described embodiment, the photographing unit 101 acquires real-time video of the site, and the display control unit 104 superimposes an electronic whiteboard image and a staff image on the video to generate a site photograph. Here, it is also possible to generate a site photograph using recorded video (video or still image) instead of the real-time video. In this case, when photographing the site video, the photographing unit 101 simultaneously acquires measurement values ​​from various sensors and stores them in association with the video (each frame in the case of a video). The display control unit 104 adjusts the scale of the staff image using the stored measurement values ​​from the various sensors. The display control unit 104 superimposes the electronic whiteboard image and the scale-adjusted staff image on the recorded video to generate a site photograph.

[0052] Each processing means constituting the present invention may be configured by hardware, or any process may be realized by having a CPU execute a computer program. Furthermore, the computer program may be stored and supplied to a computer using various types of temporary or non-temporary computer-readable media. Temporary computer-readable media include, for example, electromagnetic signals supplied to a computer via wire or wirelessly. [Explanation of symbols]

[0053] 1. Inspection system 10. First terminal device 20 Second terminal device 30 servers 101 Photography Department 102 Electronic whiteboard generation unit 103 Leveling rod generator 104 Display control unit 201 Electronic whiteboard information input section 301 Storage section

Claims

1. a photographing unit for capturing an image of a real object; a staff generator for generating one or more staff images for comparison with the object; a display control unit that displays the staff image superimposed on the video, The display control unit determining the placement of the staff image based on dimension lines included in the drawing of the object; adjusting the scale of the staff image to be the same size as the actual object; The staff image is superimposed on the video. Inspection system.

2. An electronic whiteboard generating unit generates an electronic whiteboard image including information required for the on-site photograph, the electronic whiteboard includes the drawing of the object; The display control unit determines the layout of the scale image based on the dimension lines included in the drawing included in the electronic whiteboard. The inspection system according to claim 1.

3. 1. A computer-implemented method comprising: An imaging step of acquiring an image of a real object; a staff generation step of generating one or more staff images for comparison with the object; a display control step of superimposing and displaying the staff image on the video, The display control step includes: determining the placement of the staff image based on dimension lines included in a drawing of the object; adjusting the scale of the staff image to be equal to that of the actual object; and a step of superimposing and displaying the staff image on the video. Inspection method.

4. A program for causing a computer to execute the method according to claim 3.

Images