Repair Support System

Abstract

To provide a repair work support system which supports repair work of a movable body having a sensor that acquires external world information via a transmission region set in an exterior member and allows a worker to easily grasp the transmission region.SOLUTION: A repair support system 1 for supporting repair of a movable body 3 comprising a sensor 5 that acquires external world information via a transmission region 8 set in an exterior member 7 has: an image capture device 27 which acquires a photographed image 28 of an outer surface of the movable body; a display device 29 which superimposes an image showing a specific region of the outer surface on an image 3A of the movable body and can present the images to a worker; storage devices 25, 37 which store specific information 38, 38A, 38B, 38C for specifying a portion corresponding to the transmission region from the photographed image; and processors 21, 33 which process the photographed image. The processor specifies the portion corresponding to the transmission region from the photographed image on the basis of the specific information, and causes the display device to display a portion 8A corresponding to the transmission region so as to be superimposed on the image 3A of the movable body.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a repair assistance system that assists repair work on a mobile object, particularly a vehicle. [Background technology]

[0002] A vehicle repair assistance system that can reduce the burden on a worker who repairs a vehicle while enabling the worker to properly repair the vehicle is known (for example, Patent Document 1). The vehicle repair assistance system of Patent Document 1 includes a communication terminal that acquires vehicle information from the vehicle, and a server that receives the vehicle information from the communication terminal and assists the worker in repairing the vehicle.

[0003] The server queries the repair requester about the vehicle's condition, including product classification, symptoms, detailed symptoms, a summary of the problem, the environment, and the circumstances at the time of the problem. Based on the results of the query, the server searches for information about similar cases with symptoms similar to those of the vehicle to be repaired, and notifies the communication terminal of the search results. This allows the worker to grasp detailed information about the selected similar case through the communication terminal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-225216 Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is a desire to realize a sustainable transportation system that takes into consideration the vulnerable among transportation participants, and therefore vehicles are being equipped with various sensors such as radar and lidar to further improve traffic safety.

[0006] However, there are cases where the sensor is covered by an exterior member and acquires external information through a transparent area set in the exterior member. For example, if the sensor is a radar, such a transparent area is defined as the area through which radio waves emitted from the radar pass. Therefore, workers cannot visually determine the location or area of ​​the transparent area. Repairing such a transparent area using a method other than the specified method may impair the function of the sensor, so there is a strong demand for the development of technology that allows workers to easily determine the transparent area.

[0007] In view of the above background, the present invention provides a repair work assistance system for assisting repair work on a mobile object equipped with a sensor that acquires external information through a transparent area set in an exterior member, and an object of the present invention is to enable a worker to easily grasp the transparent area, thereby contributing to the development of a sustainable transportation system by improving the safety of the mobile object. [Means for solving the problem]

[0008] In order to solve the above problem, one aspect of the present invention is a repair support system (1, 51) for supporting the repair of a mobile body (3) equipped with a sensor (5) that acquires external information through a transparent area set in an exterior member (7), the system comprising: an imaging device (27) that acquires an image (28) of the outer surface of the mobile body; a display device (29, 69) that can present to a worker an image (3A) of the mobile body by superimposing an image showing a specific area of ​​the outer surface; a storage device (25, 37) that stores specific information (38, 38A, 38B, 38C) for identifying a portion of the captured image that corresponds to the transparent area; and a processor (21, 33) that processes the captured image, wherein the processor identifies a portion of the captured image that corresponds to the transparent area based on the specific information, and causes the display device to display the portion (8A) corresponding to the transparent area superimposed on the image (3A) of the mobile body.

[0009] According to this aspect, the portion corresponding to the transparent region is superimposed on the image of the moving object and displayed on the display device, so that even if the sensor is covered by an exterior member and not directly exposed to the outside, the worker can easily grasp the position and range of the transparent region of the exterior member.

[0010] In the above aspect, the display device (29) preferably displays the captured image by superimposing the portion corresponding to the transparent region on the captured image.

[0011] According to this aspect, the worker can easily grasp the position and range of the transmissive region by visually checking the display device.

[0012] In the above aspect, preferably, the display device (69) is configured to be able to superimpose information onto a transparent lens, and the display device displays the image of the moving object visible through the lens by superimposing the portion corresponding to the transparent area.

[0013] According to this aspect, by visually observing the moving object through the lens, the worker can easily grasp the position and range of the transmission area.

[0014] In the above aspect, preferably, the processor extracts an image of an abnormal area that may reduce the detection sensitivity of the sensor from the captured image, and the display device displays an image of the moving object by superimposing a portion corresponding to the transparent area and an icon (39) indicating the abnormal area.

[0015] According to this aspect, the operator can easily find the abnormality.

[0016] In the above aspect, preferably, the processor identifies an abnormality factor corresponding to the abnormal location based on the captured image, and the display device displays the abnormality factor.

[0017] According to this aspect, the operator can easily grasp the cause of the abnormality.

[0018] In the above aspect, the abnormality factors preferably include scratches and dirt.

[0019] According to this aspect, it is possible to appropriately acquire an abnormality location that may reduce the detection sensitivity of the sensor.

[0020] In the above aspect, preferably, the processor determines whether the abnormal part can be repaired based on the captured image, and when it is determined that the abnormal part cannot be repaired, the display device notifies the worker (45) that replacement is necessary, and when it is determined that the abnormal part can be repaired, the display device notifies the worker (43) that repair is instructed.

[0021] According to this aspect, the efficiency of repair work is improved.

[0022] In the above aspect, the display device preferably updates and displays the portion corresponding to the transparent region in accordance with updating of the captured image.

[0023] According to this aspect, even if the worker moves while capturing images, the display device updates and displays the portion corresponding to the transparent area in real time, allowing the worker to more easily understand the position and range of the transparent area and more accurately grasp it.

[0024] In the above aspect, the processor preferably identifies the portion corresponding to the transmissive region based on the surface shape of the exterior member.

[0025] According to this aspect, the portion corresponding to the transmission area can be identified more accurately than when it is not based on the surface shape of the exterior member.

[0026] In the above aspect, preferably, when the processor cannot identify a portion corresponding to the transparent region from within the captured image, it issues a notification (40) to prompt the operator to take an image.

[0027] According to this aspect, when the portion corresponding to the transparent region cannot be identified from the captured image that has already been acquired, the worker can be prompted to capture an image that will enable identification.

[0028] In the above aspect, the processor preferably identifies the portion corresponding to the transmission area based on the positions of a plurality of members mounted on the moving body in addition to the exterior member.

[0029] According to this aspect, the portion corresponding to the transmissive region can be identified more accurately. [Effects of the Invention]

[0030] According to the above aspects, a repair work support system is provided that supports repair work on a mobile object equipped with a sensor that acquires external information through a transparent area set in an exterior member, and allows a worker to easily grasp the transparent area. [Brief explanation of the drawings]

[0031] [Figure 1] Schematic diagram showing an overview of a repair assistance system according to a first embodiment. [Figure 2] Repair support system block diagram [Figure 3] An example of a screen display on a terminal in a repair support system showing (A) a notification of a repairable abnormality on a mobile device and (B) a notification of an unrepairable abnormality on the mobile device. [Figure 4] Vehicle information table example [Figure 5] Flowchart of abnormality identification process [Figure 6] Flowchart of area data generation process [Figure 7] Example of a screen display with arrows indicating the direction of movement [Figure 8] FIG. 10 is a perspective view of a terminal of a repair support system according to a second embodiment; [Figure 9] Example of screen display of a terminal of the repair support system according to the second embodiment DETAILED DESCRIPTION OF THE INVENTION

[0032] Hereinafter, with reference to the drawings, an embodiment in which the mobile body repair assistance system of the present invention is applied to a system for assisting repair of a vehicle, that is, a vehicle repair assistance system, will be described.

[0033] <<First Embodiment>> A vehicle (hereinafter referred to as vehicle to be repaired 3) that is repaired by a worker supported by vehicle repair support system 1 is a four-wheeled automobile and is equipped with sensor 5. Sensor 5 acquires external information such as the position, size, direction, and distance of objects around vehicle to be repaired 3 relative to vehicle to be repaired 3. Sensor 5 may be, for example, a radar that emits radio waves toward a predetermined irradiation area 6 (also referred to as a detection area) and measures the radio waves reflected from objects located within irradiation area 6 and incident on sensor 5, thereby detecting the position and size of objects within irradiation area 6.

[0034] As shown in Fig. 1, an exterior member 7 may be provided on the vehicle exterior side of the sensor 5. In this embodiment, a bumper is provided as the exterior member 7 on the vehicle exterior side of the sensor 5. In this case, the sensor 5 acquires external information through a transmission area 8 set in the exterior member 7. The transmission area 8 corresponds to the overlapping portion between the irradiation area 6 from which the sensor 5 emits radio waves (or the area where radio waves are incident on the sensor 5) and the exterior member 7.

[0035] If an exterior member 7 is provided on the exterior side of the sensor 5 so that the sensor 5 is covered by the exterior member 7, the worker cannot easily grasp the position of the sensor 5 or the transmission area 8.

[0036] When repairing a vehicle 3 to be repaired in which the sensor 5 is covered by an exterior member 7, repairing the transparent area 8 in a manner other than that specified could impair the function of the sensor 5, so the worker needs to accurately understand the transparent area 8. The vehicle repair assistance system 1 is a system that assists workers who repair a vehicle 3 to be repaired that is equipped with a sensor 5 that acquires external information through the transparent area 8 by making the transparent area 8 easier for workers to understand.

[0037] As shown in FIGS. 1 and 2, the vehicle repair assistance system 1 includes a terminal 11 and a server 13.

[0038] 2, the terminal 11 includes a processor (CPU, central processing unit, hereinafter referred to as terminal processor 21), a main memory 23 such as ROM and RAM, a storage device 25 (flash memory, etc.), an imaging device 27, and a touch panel 29. The terminal 11 is configured to be able to communicate with the server 13 via a network 31 such as the Internet. The terminal 11 may be a so-called smartphone or a tablet terminal.

[0039] The imaging device 27 includes a single or multiple cameras 27A. As shown in Figures 3(A) and (B), the imaging device 27 captures an image of the exterior of the vehicle 3 to be repaired, thereby obtaining an image (captured image 28) including an image 3A of the vehicle 3 to be repaired. In this embodiment, the imaging device 27 is configured with multiple cameras 27A (compound eye cameras).

[0040] The touch panel 29 functions as an input / output device that includes a display device that displays images, text, etc., and an input device that accepts input. The touch panel 29 can display predetermined patterns and text superimposed on the captured image 28 acquired by the imaging device 27. For example, when the vehicle 3 to be repaired is reflected in the captured image 28, the touch panel 29 can superimpose an image showing a specific area on the exterior of the vehicle 3 to be repaired on the image 3A of the vehicle 3 to be repaired that is reflected in the captured image 28, and present the position and size of that area to the worker.

[0041] The processor of the terminal 11 (hereinafter, terminal processor 21) can display input fields and the like on the touch panel 29 and accept input from the user.

[0042] The terminal processor 21 displays an input field on the touch panel 29 and receives information related to the vehicle 3 to be repaired from the worker. The information received by the terminal processor 21 from the worker (hereinafter referred to as repair vehicle information) includes the vehicle name (the vehicle's unique name), model, etc. of the vehicle 3 to be repaired. The repair vehicle information may also include the manufacturer's name, vehicle number, registration date, etc. The terminal processor 21 transmits the received repair vehicle information to the server 13.

[0043] The terminal processor 21 transmits the captured image 28 acquired by the imaging device 27 to the server 13. In this embodiment, the terminal processor 21 is configured to be able to transmit the captured image 28 acquired by the imaging device 27 to the server 13 in real time. In addition, the terminal processor 21 can display the image and text data transmitted from the server 13 on the touch panel 29. The terminal processor 21 may have a function of using three-dimensional data transmitted from the server 13 to superimpose it on the captured image 28 and display it.

[0044] As shown in Figure 2, the server 13 is a computer equipped with a processor (CPU, MPU, etc., also called an arithmetic processing unit; hereinafter referred to as server processor 33), a main memory 35 such as ROM, RAM, etc., and a storage device (HDD or SSD, also called storage; hereinafter referred to as server storage device 37).

[0045] The server storage device 37 stores identification information 38 for identifying a portion of the captured image 28 that corresponds to the transparent area 8. The identification information 38 includes a vehicle information table 38A, a vehicle appearance data group 38B, and a transparent area data group 38C. As shown in Fig. 4, the vehicle information table 38A stores the vehicle name, model, vehicle appearance data name, and overlap area data name in association with each other.

[0046] The vehicle appearance data group 38B includes vehicle appearance data. The vehicle appearance data is data for estimating the position and attitude of the image capturing device 27 from an image captured by the image capturing device 27 and determining the position and attitude of the terminal 11. The vehicle appearance data may be, for example, data indicating the outer surface of the vehicle with the center or center of gravity of the vehicle as the origin, or may be data indicating the positions of a plurality of characteristic points.

[0047] The transparent area data group 38C includes transparent area data. The transparent area data is data indicating the irradiation range of the bumper (exterior member 7) through which radio waves emitted from the sensor 5 mounted on the vehicle penetrate, for a vehicle whose name and model correspond to the transparent area data name in the vehicle information table 38A. In other words, the transparent area data is data indicating the overlapping portion of the detection area of ​​the sensor 5 and the bumper, i.e., the position, size, and range of the transparent area 8. The server storage device 37 stores the transparent area data as data (model data) including a three-dimensional point sequence and indicating the shape of an object in three-dimensional space. However, the server storage device 37 may also store the transparent area data as data indicating a point cloud constituting the outer surface of the overlapping portion of the detection area of ​​the sensor 5 and the bumper.

[0048] When the server processor 33 acquires the repair vehicle information from the terminal 11, it extracts the vehicle name and model included in the repair vehicle information. Then, the server processor 33 references the vehicle information table 38A stored in the server storage device 37 and extracts the corresponding transparent area data name. Then, the server processor 33 identifies the transparent area data corresponding to the transparent area data name from the transparent area data group 38C, i.e., the transparent area data indicating the transparent area 8 set in the exterior member 7 of the vehicle 3 to be repaired.

[0049] Furthermore, when the server processor 33 acquires the repair vehicle information from the terminal 11, it refers to the vehicle information table 38A and acquires the vehicle appearance data corresponding to the vehicle 3 to be repaired.

[0050] Thereafter, when the server processor 33 acquires the captured image 28 from the terminal 11, the server processor 33 executes, in parallel, an abnormality identification process for processing the captured image 28 to identify an abnormal location, an abnormality cause, etc., and an area data generation process for processing the captured image 28 to superimpose an image showing the transparent area 8 set in the exterior member 7 on the captured image 28. Hereinafter, the abnormality identification process will first be described in detail with reference to FIG. 5.

[0051] In the first step ST1 of the anomaly identification process, the server processor 33 determines whether or not there are any abnormalities, such as scratches or stains on the vehicle body, from the captured image 28 acquired by the imaging device 27. The server processor 33 may determine the presence or absence of abnormalities by extracting abnormalities from the image captured by the imaging device 27 based on a known machine learning method using a constructed learning model. The server processor 33 may perform enhancement processing, filtering processing, etc. on the image captured by the imaging device 27, and extract abnormalities based on, for example, shading, shape, etc., to determine the presence or absence of abnormalities.

[0052] If the server processor 33 cannot identify an abnormality from the captured image 28 acquired by the imaging device 27, it may transmit a request signal to the terminal 11 requesting further images. When the terminal 11 receives the request signal, the terminal processor 21 displays on the touch panel 29 a notice urging the worker to capture another image using the imaging device 27. At this time, the touch panel 29 of the terminal 11 may display, for example, "The image is unclear. Please take another image." When the worker captures another image using the imaging device 27, the terminal processor 21 transmits the captured image 28 acquired by the imaging device 27 to the server processor 33. Thereafter, the server processor 33 determines whether or not there is an abnormality based on the newly transmitted captured image 28.

[0053] If an abnormality is found, the server processor 33 proceeds to step ST2, and if an abnormality is not found, the server processor 33 ends the abnormality identification process.

[0054] In step ST2, the server processor 33 identifies the position of the abnormality and generates data (hereinafter, abnormality position data) for displaying an icon 39 indicating the position and size of the abnormality on the touch panel 29 of the terminal 11. The icon 39 may be, for example, a frame surrounding the abnormality, and the abnormality position data may be numerical data indicating the shape and position of the frame corresponding to the icon 39 in the image captured by the imaging device 27. Once acquisition of the abnormality position data is complete, the server processor 33 executes step ST3.

[0055] In step ST3, the server processor 33 analyzes the captured images 28 acquired by the imaging device 27 and identifies the abnormality factors corresponding to each abnormal location. The abnormality factors include scratches, dirt, etc. The server processor 33 may identify the abnormality factors for each abnormal location based on a known machine learning method using a pre-constructed learning model. Alternatively, the server processor 33 may perform enhancement processing or filtering on the images acquired by the imaging device 27 and identify the factors corresponding to the abnormal locations based on shading, shape, etc. The server processor 33 identifies the abnormality factors for each abnormal location and acquires abnormality factor data indicating the abnormality factors for each abnormal location.

[0056] In this embodiment, when the abnormal portion is a scratch or stain, the server processor 33 calculates a level (hereinafter, abnormality level) indicating the degree of the scratch or stain (for example, the depth of the scratch or the ease of removing the stain).

[0057] When the acquisition of the cause and level of the abnormality for each abnormal location is completed, the server processor 33 executes step ST4.

[0058] In step ST4, the server processor 33 extracts an abnormal location whose cause of abnormality is a defect, and if an abnormal location whose cause of abnormality is a defect is extracted, the server processor 33 determines whether or not the abnormal location can be repaired. For example, when an abnormal location whose cause of abnormality is a defect is extracted, the server processor 33 may determine whether or not the abnormal location can be repaired based on the abnormality level of the abnormal location. The server processor 33 acquires data indicating whether or not the abnormal location whose cause of abnormality is a defect can be repaired (hereinafter, repairability data). Once acquisition of the repairability data has been completed, the server processor 33 executes step ST5.

[0059] In step ST5, the server processor 33 transmits the abnormality location data, the abnormality cause data, and the repair possibility data to the terminal 11. When the transmission is completed, the server processor 33 ends the abnormality identification process.

[0060] Next, the area data generation process will be described in detail with reference to FIG.

[0061] In step ST11, the server processor 33 identifies the position and attitude of the terminal 11 relative to the vehicle 3 to be repaired. In this embodiment, the server processor 33 identifies the position and attitude of the terminal 11 based on the vehicle appearance data and the captured image 28 acquired by the imaging device 27.

[0062] Specifically, the server processor 33, for example, determines an image of the exterior member 7 of the vehicle 3 to be repaired from an image acquired by the imaging device 27, and acquires its surface shape. Thereafter, the server processor 33 identifies the position and attitude of the terminal 11 by comparing the acquired surface shape with the vehicle appearance data.

[0063] Additionally, the server processor 33 may determine an image of the vehicle 3 to be repaired and extract feature points in the image. The server processor 33 may then identify the position and attitude of the terminal 11 by comparing the positions of the acquired feature points with the positions of feature points included in the vehicle appearance data. The server processor 33 may identify the position and attitude of the terminal 11 based on the positions of multiple components mounted on the vehicle 3 to be repaired in addition to the exterior component 7, as appropriate.

[0064] In this embodiment, if the server processor 33 cannot determine the position and orientation of the terminal 11 from the image acquired by the imaging device 27, it transmits a request signal to the terminal 11 requesting further images. When the terminal 11 receives the request signal, the terminal processor 21 displays on the touch panel 29 a notification 40 prompting the terminal 11 to take further images using the imaging device 27. At this time, the touch panel 29 of the terminal 11 may display, for example, a message saying "Please take a photo of the car to be repaired from a different angle," as shown in FIG. 7(A). When the worker again captures an image using the imaging device 27, the terminal processor 21 transmits the captured image to the server processor 33. The server processor 33 then determines the position and orientation of the terminal 11 based on the newly transmitted image.

[0065] When the position and attitude of the terminal 11 have been specified, the server processor 33 executes step ST12.

[0066] In step ST12, the server processor 33 reads the transparent area data identified based on the repair vehicle information. Thereafter, the server processor 33 determines whether the transparent area 8 is located within the range imaged by the imaging device 27, based on the position and attitude of the terminal 11 identified in step ST11. If the transparent area 8 is located within the range imaged by the imaging device 27, step ST13 is executed, and if the transparent area 8 is not located within the range, step ST14 is executed.

[0067] In step ST13, the server processor 33 generates an image (hereinafter, "transparent area image 8A") showing a portion corresponding to the transparent area 8 in the captured image 28 acquired by the imaging device 27, based on the position and orientation of the terminal 11 and the transparent area data. The transparent area image 8A may be frame-shaped, and the inside of the frame may be colored. The server processor 33 may generate the transparent area image 8A as data of points for constituting a frame corresponding to the transparent area image 8A. When the generation of the transparent area image 8A is completed, the server processor 33 executes step ST15.

[0068] In step ST14, the server processor 33 acquires data (hereinafter, movement direction data) indicating the movement direction of the terminal 11 required to include the transparent area 8 within the imaging area of ​​the imaging device 27. The server processor 33 may acquire the movement direction data based on the position and posture of the terminal 11 and the transparent area data. Once acquisition of the movement direction data is complete, the server processor 33 transmits the movement direction data to the terminal 11. Once transmission of the movement direction data is complete, the server processor 33 ends the area data generation process.

[0069] In step ST15, the server processor 33 transmits the transmission area image 8A to the terminal 11. When the transmission is completed, the server processor 33 ends the area data generation process.

[0070] When the terminal processor 21 receives the abnormality location data, the abnormality cause data, and the repair feasibility data, the terminal processor 21 superimposes an icon 39 indicating the abnormality location on an image captured by the imaging device 27, particularly an image 3A of the vehicle 3 to be repaired, and displays the image on the touch panel 29 based on the abnormality location data. The terminal processor 21 acquires the abnormality cause corresponding to each of the abnormality locations displayed on the touch panel 29 based on the abnormality cause data, and displays information indicating the abnormality cause on the touch panel 29 using text 41, a design, or the like. Furthermore, the terminal processor 21 displays whether each of the abnormality locations displayed on the touch panel 29, where the abnormality cause is a scratch, can be repaired based on the repair feasibility data. In this embodiment, for example, for an abnormality location that can be repaired, the terminal processor 21 displays a notification 43 instructing repair (also referred to as repair), as shown in FIG. 3(A). For an abnormality location that cannot be repaired, the terminal processor 21 displays a notification 45 on the touch panel 29 indicating that replacement is required, as shown in FIG. 3(B). When detecting contact with an icon 39 indicating an abnormality displayed on the touch panel 29, the terminal processor 21 may display a corresponding notification 43, 45. Furthermore, as shown in Fig. 3(A), when the terminal processor 21 displays notification 43 instructing repair, it may also display a repair method (maintenance method).

[0071] When the terminal processor 21 acquires the transmission area image 8A, it generates a composite image by superimposing a frame corresponding to the transmission area image 8A on the captured image 28 acquired by the imaging device 27, and displays the composite image on the touch panel 29. When acquiring abnormal area position data, the terminal processor 21 may simultaneously superimpose a frame corresponding to the transmission area image 8A and an icon 39 indicating the abnormal area on the captured image 28 acquired by the imaging device 27 and display them on the touch panel 29.

[0072] As shown in Fig. 7(B), when the terminal processor 21 acquires movement direction data, it issues a notification 47 on the touch panel 29 indicating the movement direction corresponding to the movement direction data. The notification 47 indicating the movement direction may be, for example, an arrow indicating the movement direction corresponding to the movement direction data. In this embodiment, the terminal processor 21 may superimpose an arrow indicating the movement direction on the captured image 28 together with the text "Please move in the direction of the arrow."

[0073] Next, the operation and effects of the vehicle repair assistance system 1 configured as above will be described.

[0074] When a worker holds the terminal 11, takes an image of the vehicle 3 to be repaired, and transmits the image to the server 13, the server processor 33 starts an abnormality identification process and an area data generation process.

[0075] In the abnormality identification process, the server processor 33 determines whether or not there are any abnormalities such as scratches or dirt on the vehicle body from the images acquired by the imaging device 27, and if there are any abnormalities, it transmits abnormality location data, abnormality cause data, and repair possibility data to the terminal 11.

[0076] Furthermore, the server processor 33 identifies the position and orientation of the terminal 11 in the area data generation process, and transmits a transparent area image 8A to the terminal 11 if the transparent area 8 is included in the imaging area of ​​the imaging device 27.

[0077] 3A and 3B, upon receiving the abnormality location data, abnormality cause data, repairability data, and transparent area image 8A, the terminal 11 (terminal processor 21) superimposes an icon 39 indicating the abnormality location and the transparent area image 8A on the image acquired by the imaging device 27, and displays the superimposed image on the touch panel 29. That is, the server processor 33 and the terminal processor 21 cooperate to identify the portion of the captured image 28 that corresponds to the transparent area 8 based on the identification information 38, and superimpose the transparent area image 8A, which is the portion that corresponds to the transparent area 8, on the image 3A of the vehicle 3 to be repaired in the captured image 28, and display the superimposed image on the touch panel 29.

[0078] As a result, an image of the transparent area 8 (transparent area image 8A), which cannot be seen with the naked eye in the real world, is superimposed and displayed on the touch panel 29 on the image 3A of the vehicle 3 to be repaired in the captured image 28. Therefore, even if the sensor 5 is covered by the exterior member 7 and is not directly exposed to the outside, the worker can easily understand the part of the exterior member 7 that is within the detection range of the sensor 5 by visually checking the display on the touch panel 29.

[0079] In this way, the vehicle repair assistance system 1 uses virtual reality technology (AR (Augmented Reality) technology) to superimpose an area that cannot be seen in the real world on the captured image 28 and display it on the touch panel 29. This allows the worker to grasp the transparent area 8 without having to manually mark the vehicle 3 to be repaired to indicate the irradiation range of the radar radio waves (transparent area 8). This makes it possible to significantly reduce the time required for repair work. Furthermore, because the transparent area 8 is displayed by calculation processing, it is possible to present the worker with a more accurate transparent area 8 than if it were marked manually. Furthermore, because it is configured using a smartphone, the vehicle repair assistance system 1 can be configured relatively inexpensively without using a separate imaging device 27 or the like.

[0080] Furthermore, the server processor 33 extracts from the captured image 28 an image of an abnormal portion that may reduce the detection sensitivity of the sensor 5, and an icon 39 indicating the abnormal portion is displayed on the touch panel 29 superimposed on the captured image 28. Therefore, the worker can easily identify the abnormal portion by checking the display on the touch panel 29.

[0081] Furthermore, the server processor 33 acquires the cause of the abnormality corresponding to the abnormal location from the captured image 28, and displays information (text 41) indicating the cause of the abnormality on the touch panel 29. The worker can easily understand the cause of the abnormality at the abnormal location from the display on the touch panel 29. This reduces the burden on the worker performing the repair. Furthermore, since the cause of the abnormality includes scratches and dirt, the worker can understand whether the cause of the abnormality at the abnormal location is scratches or dirt. This allows the worker to understand the abnormal location, such as scratches or dirt that reduces the detection sensitivity of the sensor 5, from the display on the touch panel 29, and can take appropriate action to address the abnormal location.

[0082] The terminal processor 21 displays text 41 indicating the cause of the abnormality, as well as whether or not each abnormal part whose cause is a scratch can be repaired, on the touch panel 29. The worker can understand whether or not the part can be repaired from the display on the touch panel 29, which improves the efficiency of the repair work.

[0083] The terminal processor 21 displays a notice 43 instructing repair for an abnormal part that can be repaired, and displays a notice that replacement is necessary for an abnormal part that cannot be repaired on the touch panel 29. This allows the worker to easily understand the work that needs to be done, further improving work efficiency.

[0084] The terminal processor 21 is configured to be able to transmit images captured by the imaging device 27 to the server 13 in real time. Therefore, when the worker holds the terminal 11 and moves while taking images, the terminal processor 21 transmits the images captured by the imaging device 27 to the server 13 in accordance with the worker's movement. When the server 13 acquires the images from the terminal 11, the server processor 33 transmits abnormality location data, abnormality cause data, repairability data, and transmission area image 8A to the terminal 11 in real time.

[0085] As a result, the touch panel 29 of the terminal 11 displays the transparent region image 8A and the icon 39 indicating the abnormality superimposed on the captured image 28 captured by the imaging device 27 in real time. Therefore, even if the worker moves while capturing an image, the transparent region image 8A and the icon 39 indicating the abnormality are updated and displayed in real time on the touch panel 29, allowing the worker to more easily understand the position and range of the transparent region 8. Furthermore, the worker can check the transparent region image 8A from various directions, allowing the worker to more accurately grasp the transparent region 8.

[0086] 7(B), when the terminal processor 21 acquires the movement direction data, it displays the movement direction corresponding to the movement direction data on the touch panel 29. This allows the worker to grasp the position and direction of the sensor 5, and easily understand the places on the exterior of the vehicle 3 to be repaired that need attention when carrying out repair work.

[0087] Furthermore, when the server processor 33 cannot determine the presence or absence of an abnormality from the image acquired by the imaging device 27 or cannot grasp the position and attitude of the terminal 11, a notification 47 urging the operator to re-image the image is displayed on the touch panel 29 of the terminal 11. As a result, even when the image is unclear and the abnormality cannot be identified, or when the position and attitude of the terminal 11 cannot be identified and a transmission area image 8A corresponding to the overlapping portion cannot be acquired, the operator can re-image the image and transmit it, making it possible to identify the abnormality or overlapping portion.

[0088] In this way, the server processor 33 identifies the position and attitude of the terminal 11 by comparing the acquired surface shape with the vehicle appearance data. This allows the transparent area 8 to be identified more accurately than when it is not based on the surface shape of the exterior member 7. Furthermore, the server processor 33 acquires the position and attitude of the terminal 11 based on the positions of multiple components mounted on the vehicle 3 to be repaired, in addition to the exterior member 7, as appropriate, and identifies the transparent area 8. This allows the transparent area 8 to be identified more accurately.

[0089] <<Second embodiment>> The vehicle repair assistance system 51 according to the second embodiment differs from the first embodiment in the configuration of the terminal 61. The other configurations are the same as those of the first embodiment, so a description thereof will be omitted. However, in the following description, it is assumed that the repair vehicle information relating to the vehicle 3 to be repaired is separately acquired by the server 13.

[0090] As shown in FIG. 8, the terminal 61 in the second embodiment is a so-called terminal that can be worn by a worker, that is, a wearable terminal. In this embodiment, the terminal 61 is an eyeglass-type wearable terminal (AR wearable device) that uses augmented reality (AR) technology. The terminal 61 is configured as so-called smart glasses. The terminal 61 includes a frame 63 that is fastened to the ears of the worker, and two lenses 65 (also called glasses) that are attached to the frame 63 and placed in front of the eyes of the worker wearing the terminal 61.

[0091] The terminal 61 includes an imaging device 27 , a display device 69 , a speaker 71 , and a microcomputer 73 .

[0092] The imaging device 27 is provided on the frame 63 and is composed of a camera 27A that captures an image in front of the worker wearing the terminal 61. The display device 69 is a device that can semi-transparently display information such as images and text within the worker's field of vision. In this embodiment, the display device 69 displays images and text on the lens 65 in a semi-transparent state, superimposed on a background (i.e., overlay display). The speaker 71 notifies the worker wearing the terminal 61 by voice.

[0093] Each microcomputer 73 includes a processor (terminal processor 21), a main memory 23, and a storage device 25, similar to those in the first embodiment. The microcomputer 73 is configured to be able to communicate with the server 13 via the network 31. In this embodiment, the microcomputer 73 is provided in the frame 63. However, this is not limiting, and the microcomputer 73 may be connected to the imaging device 27 and the display device 69 via cables connected to the frame 63.

[0094] The processor of the microcomputer 73 (hereinafter referred to as the terminal processor 21) transmits the image captured by the imaging device 27 to the server 13, similarly to the first embodiment.

[0095] Each server 13 includes a processor (server processor 33), a main memory 35, and a storage device (server storage device 37), similar to those in the first embodiment. The processor of the server 13 (hereinafter referred to as server processor 33) performs anomaly identification processing and area data generation processing, similar to those in the first embodiment, and when an abnormality location and a transmission area image 8A are acquired, transmits the abnormality location position data, abnormality cause data, repairability data, and the transmission area image 8A.

[0096] 9, the terminal processor 21 displays, on the display device 69, the transmission area image 8A and an icon 39 indicating the abnormality location superimposed on the image of the vehicle 3 to be repaired that is visible through the lens 65. Additionally, the terminal processor 21 may superimpose on the display device 69 the cause of the abnormality corresponding to the abnormality location and whether or not the vehicle can be repaired, as in the first embodiment.

[0097] When the terminal processor 21 receives a request signal requesting further images, it may output a voice message such as "Please move" from the speaker 71 to prompt the imaging device 27 to take further images from another position.

[0098] Next, the effects of the vehicle repair assistance system 51 configured as above will be described.

[0099] The lens 65 displays an image 3A of the vehicle 3 to be repaired, which is visible through the lens 65, with a transmission area image 8A and an icon 39 indicating the abnormality area superimposed on it. As a result, even if the sensor 5 is covered by the exterior member 7 and not directly exposed to the outside, the worker can easily grasp the part of the exterior member 7 that is within the detection range of the sensor 5, i.e., the transmission area 8, by looking at the vehicle 3 to be repaired through the lens 65. In particular, in this embodiment, the terminal 61 is a glasses-type wearable terminal, so the worker can perform repair work with both hands while looking at the transmission area image 8A and the icon 39 indicating the abnormality area, which is expected to further improve work efficiency.

[0100] When the terminal processor 21 receives a request signal requesting further images, it issues a notification urging the imaging device 27 to capture an image from another position. As a result, even if it is difficult for the imaging device 27 to capture the abnormal area or the position and attitude of the terminal 61, the terminal 61 is moved, and the server processor 33 can acquire the abnormal area or the position and attitude of the terminal 61.

[0101] In this way, the vehicle repair assistance systems 1, 51 according to the present invention improve the efficiency of repair work by workers and further enhance the safety of moving objects (vehicles). Therefore, the present invention is expected to contribute to the development of sustainable transportation systems.

[0102] Although the description of the specific embodiment has been completed above, the present invention is not limited to the above embodiment and its modifications, and can be modified in a wide range of ways.

[0103] In the above embodiment, an example has been described in which the moving body is a four-wheeled automobile (vehicle), but the moving body is not limited to a four-wheeled automobile. The moving body may be, for example, a two-wheeled vehicle, a three-wheeled vehicle, a robot, a drone, a work machine, a railroad vehicle, a ship, an airplane, etc.

[0104] In the above embodiment, the sensor 5 is a radar, but is not limited to this and may be any device that transmits and receives radio waves, sound waves, light, etc. For example, the sensor 5 may be a sonar that is covered by an exterior member 7 such as a bumper and transmits and receives sound waves to the outside of the vehicle.

[0105] In the first embodiment, the terminal processor 21 generates a composite image by superimposing the transparent region image 8A on the image acquired by the imaging device 27, but the present invention is not limited to this. The server processor 33 may superimpose the transparent region image 8A on the image acquired by the imaging device 27 and transmit the composite image to the terminal 61, and the terminal processor 21 may display the composite image transmitted from the server 13 on the touch panel 29.

[0106] In the above embodiment, the server storage device 37 stores the specific information 38, but the present invention is not limited to this, and the storage device 25 of the terminal 11 may store the specific information 38. Alternatively, the server storage device 37 and the storage device 25 of the terminal 11 may each store the specific information 38 separately. Alternatively, some of the specific information 38 may be stored by multiple servers other than the server 13, and the remaining information of the specific information 38 may be stored by the server storage device 37 and the storage device 25 of the terminal 11.

[0107] In the first embodiment, the captured image 28, the transmission area image 8A, and the icon 39 indicating the abnormality area are superimposed and displayed in real time on the touch panel 29 of the terminal 61, but the present invention is not limited to the superimposed display in real time. Alternatively, the terminal 11 may superimpose and display the transmission area image 8A on the captured image 28 after a predetermined time has elapsed since the imaging device 27 acquired the captured image 28.

[0108] Furthermore, when the server processor 33 determines that the captured image 28 transmitted in real time contains a transparent region 8, the server processor 33 may be configured to display a notification on the terminal 11 that a still image will be captured. The notification that a still image will be captured may be any notification that prompts the worker to take a photo, and may be, for example, a text message such as "Please take a photo." After that, when the captured image 28 is transmitted from the terminal 11 to the server 13, the server 13 may transmit the transparent region image 8A, and the terminal 11 may display the transparent region image 8A superimposed on the captured image 28.

[0109] Alternatively, when the terminal processor 21 determines that the abnormal part can be repaired, it may send a signal to the server 13 requesting an explanation of the repair method. The server 13 may send text or an image indicating the corresponding repair method to the terminal 61, and the terminal processor 21 may display the acquired text or image indicating the repair method on the touch panel 29.

[0110] In the first embodiment described above, the server processor 33 is configured to acquire data indicating the movement direction of the terminal 61 in step ST14. However, in addition to the movement direction of the terminal 61, the server processor 33 may also acquire the rotation direction (direction of turning) and movement distance of the terminal 61 required to include the transparent area 8 within the imaging area of ​​the imaging device 27. In this case, the server processor 33 may transmit displacement data relating to the movement direction, rotation direction, and movement distance to the terminal 61. Upon receiving the displacement data, the terminal 61 may display the corresponding movement direction, rotation direction, and movement distance on the touch panel 29.

[0111] In the above embodiment, the server processor 33 is configured to extract all abnormal areas in the captured image, but this is not limited to this. The server processor 33 may extract the portion located within the transparent area 8 (i.e., overlapping with the transparent area image 8A) from the image captured by the imaging device 27, and extract the abnormal area only from the extracted portion. Since the area through which the radio waves and sound waves from the sensor 5 pass is mostly made up of relatively flat surfaces, it becomes easy to extract the abnormal area, and the amount of calculation by the server processor 33 can be reduced.

[0112] In the second embodiment, the terminal 61 is a glasses-type wearable device, but is not limited to this. For example, the terminal 61 may be configured as a contact lens-type wearable device. The terminal 61 may also be configured as a wristwatch-type wearable device (a so-called smart watch). The imaging device 27 may also be provided on a remotely operable robot. In this case, the transparent area image 8A may be displayed superimposed on the image 3A of the vehicle 3 to be repaired on a device (e.g., a touch panel 29) that presents information to a worker remotely operating the robot. [Explanation of symbols]

[0113] 1: Vehicle repair assistance system (repair assistance system according to the first embodiment) 3: Vehicle to be repaired (mobile object) 3A: Statue (Statue of the vehicle to be repaired) 5: Sensor 7: Exterior materials 8:Transparent area 8A:Transmission area image 21: Terminal processor (processor) 23: Main memory 25: Storage device 27: Imaging device 28: Captured image 29: Touch panel (display device) 33: Server processor (processor) 37: Server storage device (storage device) 38: Specific information 38A: Vehicle information table 38B: Vehicle exterior data set 38C:Transparent area data set 39: Icon 40 :Notification 43 :Notification 45 :Notification 47 :Notification 51: Vehicle repair support system (repair support system according to the second embodiment) 65: Lens 69:Display device

Claims

1. A repair assistance system for assisting repair of a mobile object having a sensor that acquires external information through a transparent area set in an exterior member, an imaging device for capturing an image of the exterior of the moving object; a display device capable of superimposing an image showing a specific area of ​​the outer surface on an image of the moving body and presenting the image to a worker; a storage device that stores identification information for identifying a portion of the captured image corresponding to the transmission region; a processor for processing the captured image, The processor identifies a portion of the captured image that corresponds to the transparent region based on the identification information, and displays the portion that corresponds to the transparent region on the display device by superimposing it on an image of the moving body.

2. The repair support system according to claim 1 , wherein the display device displays the captured image by superimposing a portion corresponding to the transparent region on the captured image.

3. the display device is configured to be able to superimpose information on a transparent lens, The repair assistance system according to claim 1 , wherein the display device displays the image of the moving object visually recognized through the lens by superimposing the portion corresponding to the transparent region on the image.

4. The processor extracts, from the captured image, an image of an abnormality that may reduce the detection sensitivity of the sensor; The repair support system according to any one of claims 1 to 3, wherein the display device displays an image of the moving body, superimposed on it, a portion corresponding to the transparent area and an icon indicating the abnormality.

5. The processor identifies an abnormality factor corresponding to the abnormal location based on the captured image, The repair support system according to claim 4 , wherein the display device displays the cause of the abnormality.

6. The repair support system according to claim 5 , wherein the abnormality factors include scratches and dirt.

7. The processor determines whether or not the abnormality can be repaired based on the captured image, A repair support system according to any one of claims 4 to 6, wherein when it is determined that the abnormal part cannot be repaired, the display device notifies the worker that replacement is necessary, and when it is determined that the abnormal part can be repaired, the display device notifies the worker that repairs are required.

8. 8. The repair support system according to claim 1, wherein the display device updates and displays the portion corresponding to the transparent region in accordance with an update of the captured image.

9. The repair support system according to any one of claims 1 to 8, wherein the processor identifies the portion corresponding to the transmission area based on the surface shape of the exterior member.

10. The repair support system according to any one of claims 1 to 9, wherein the processor notifies the worker to take an image when the portion corresponding to the transparent region cannot be identified from the captured image.

11. The repair support system according to any one of claims 1 to 10, wherein the processor identifies the portion corresponding to the transparent area based on the positions of a plurality of components mounted on the moving body in addition to the exterior component.

Images