State determination device, system, state determination method and program

Abstract

To reduce a workload and improve accuracy of an inspection of a fastening member such as a nut or a bolt attached to a tire wheel of a vehicle.SOLUTION: A state determination device 20 includes a control part 21 configured to: retrieve a photographic image including a first member having a first marking and a second member having a second marking corresponding to the first marking, being a fastening member configured to fasten a wheel to a vehicle main body; calculate a first straight line passing through the first marking and a center location of the first member at the retrieved photographic image, and a second straight line passing through the second marking and the center location of the first member or a center location of the second member; and determine a fastening state of the first member according to the first straight line and the second straight line thus calculated.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a state determination device, a system, a state determination method, and a program. [Background technology]

[0002] Patent Document 1 discloses a system that calculates the rotation angle around the axis of a bolt based on an image of the bolt and determines whether the bolt is loose. Patent Document 2 discloses an apparatus that extracts a marking pattern on the bolt fastening portion from an image of the bolt and inputs the extracted marking pattern into a trained marking determination model that has learned the relationship between the marking pattern and the fastening state of the bolt through machine learning, thereby determining the fastening state of the bolt. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-127981 [Patent Document 2] Japanese Patent Application Publication No. 2023-176599 Summary of the Invention [Problem to be solved by the invention]

[0004] Drivers of large vehicles are required to check that the nuts on all wheels are not loose before driving every day.One method of visualizing loose nuts is to mark the circumference of the bolt and the nut, but to prevent accidents caused by overlooking loose nuts or insufficient inspection, there is a need for a system that can detect loose nuts more accurately and without omissions than conventional systems.

[0005] An object of the present disclosure is to reduce the effort and improve the accuracy of inspecting fastening members such as nuts or bolts attached to vehicle tire wheels. [Means for solving the problem]

[0006] Some aspects of the present disclosure are set forth below.

[0007] (1) Acquire a photographed image including a first member having a first marking, which is a fastening member for fixing a wheel to a vehicle body, and a second member having a second marking corresponding to the first marking; calculating a first straight line passing through a center position of the first member and the first marking in the acquired photographed image, and a second straight line passing through the center position of the first member or the center position of the second member and the second marking; The fastening state of the first member is determined based on the calculated first straight line and the calculated second straight line. A state determination device including a control unit. This embodiment makes it possible to reduce the labor required for inspecting fastening members and improve the accuracy.

[0008] (2) The control unit calculating an intersection angle between the first straight line and the second straight line in the captured image; The state determination device according to (1), wherein the fastening state is determined to be whether the first member is loose or not depending on whether the calculated intersection angle exceeds a threshold value. This embodiment makes it possible to detect looseness of the first member more accurately and without omission.

[0009] (3) The control unit As the second straight line, a straight line passing through the center position of the second member and the second marking in the photographed image is calculated; Detecting an outline of the first member in the captured image; The state determination device according to (2), wherein when calculating the intersection angle, the calculated value of the intersection angle is corrected depending on the shape of the detected contour. This aspect makes it less likely that erroneous detection will occur even if the image is taken at an oblique angle.

[0010] (4) The control unit As the second straight line, a straight line passing through the center position of the second member and the second marking in the photographed image is calculated; Detecting an outline of the second member in the captured image; The state determination device according to (2), wherein when calculating the intersection angle, the calculated value of the intersection angle is corrected depending on the shape of the detected contour. This aspect makes it less likely that erroneous detection will occur even if the image is taken at an oblique angle.

[0011] (5) The control unit Detecting an outline of the first member in the captured image; The state determination device according to any one of (1) to (4), wherein the center position of the first member is identified based on the detected contour. This embodiment improves the detection accuracy.

[0012] (6) The control unit weighting each coordinate included in the coordinate group of the first marking in the captured image according to a relative position; The state determination device according to any one of (1) to (5), wherein the first straight line is calculated as a straight line passing through the center position of the first member in the captured image and an average coordinate of the group of coordinates of the first marking, each of which is weighted. This embodiment improves the detection accuracy.

[0013] (7) The state determination device according to any one of (1) to (6), wherein, when the first member and the second member each have markings of multiple colors, the control unit detects markings of the same color as the first marking and the second marking. This makes it less likely that false detection will occur even if old markings remain.

[0014] (8) The state determination device according to any one of (1) to (7), wherein the first member is a nut or a bolt. This embodiment makes it possible to reduce the labor required for inspecting nuts or bolts and improve the accuracy.

[0015] (9) The state determination device according to any one of (1) to (7), wherein the first member is a nut and the second member is a bolt. This embodiment makes it possible to reduce the labor required for inspecting nuts and bolts and improve the accuracy.

[0016] (10) A state determination device according to any one of (1) to (9), an imaging device for obtaining the captured image; A system comprising: This embodiment makes it possible to reduce the labor required for inspecting fastening members and improve the accuracy.

[0017] (11) A computer acquires a photographed image including a first member having a first marking, which is a fastening member for fixing a wheel to a vehicle body, and a second member having a second marking corresponding to the first marking; The computer calculates a first line passing through a center position of the first member and the first marking in the acquired photographed image, and a second line passing through the center position of the first member or the center position of the second member and the second marking; the computer determines a fastening state of the first member based on the calculated first straight line and the calculated second straight line; A state determination method including: This embodiment makes it possible to reduce the labor required for inspecting fastening members and improve the accuracy.

[0018] (12) Acquiring a photographed image including a first member having a first marking, which is a fastening member for fixing a wheel to a vehicle body, and a second member having a second marking corresponding to the first marking; Calculating a first straight line passing through a center position of the first member and the first marking in the acquired photographed image, and a second straight line passing through the center position of the first member or the center position of the second member and the second marking; determining a fastening state of the first member based on the calculated first straight line and the calculated second straight line; A program that causes a computer to perform operations including: This embodiment makes it possible to reduce the labor required for inspecting fastening members and improve the accuracy. [Effects of the Invention]

[0019] According to the present disclosure, it is possible to reduce the labor and improve the accuracy of inspecting fastening members such as nuts or bolts attached to vehicle tire wheels. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a system according to an embodiment of the present disclosure. [Figure 2] 4 is a flowchart illustrating an operation of the state determination device according to the embodiment of the present disclosure. [Figure 3] This is an example of a full image of a wheel. [Figure 4] FIG. 10 is a diagram showing an example of calculation of a first straight line in a captured image. [Figure 5] FIG. 10 is a diagram showing an example of calculation of a second straight line in a captured image. [Figure 6] 10A and 10B are diagrams illustrating examples of the intersection angle between the first line and the second line. [Figure 7] FIG. 10 is a diagram showing an example in which a photographed image is photographed obliquely. [Figure 8] 10A and 10B are diagrams illustrating an example of correction of a calculated value of an intersection angle. [Figure 9] FIG. 10 is a diagram showing an example of a bolt having characters engraved on its head. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0022] In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0023] The configuration of a system 10 according to this embodiment will be described with reference to FIG.

[0024] The system 10 according to this embodiment includes a state determination device 20 and a terminal device 30. The state determination device 20 is capable of communicating with the terminal device 30 via a network 40.

[0025] In this embodiment, the state determination device 20 is a server computer such as a cloud server, but it may also be a general-purpose computer such as a PC, a microcomputer installed in a mobile device such as a smartphone or tablet, or a dedicated computer. "PC" is an abbreviation for personal computer. The state determination device 20 may be integrated with the terminal device 30, or may be, for example, a microcomputer installed in the terminal device 30.

[0026] The terminal device 30 is a mobile device such as a smartphone or tablet used by a user such as a driver of a large vehicle. The terminal device 30 has an imaging unit such as a camera and can function as an imaging device. The terminal device 30 has an output unit such as a display or a speaker and can function as an output device.

[0027] Network 40 may include the Internet, at least one WAN, at least one MAN, or any combination thereof. "WAN" is an abbreviation for wide area network. "MAN" is an abbreviation for metropolitan area network. Network 40 may include at least one wireless network, at least one optical network, or any combination thereof. A wireless network may be, for example, an ad-hoc network, a cellular network, a wireless LAN, a satellite communication network, or a terrestrial microwave network. "LAN" is an abbreviation for local area network.

[0028] An overview of this embodiment will be described with reference to FIG.

[0029] An image showing the entire wheel of a vehicle such as a large vehicle is captured by a camera installed at the entrance / exit of a parking lot, yard, or warehouse, or by a camera on the terminal device 30. In the case of a stationary camera, a switch or sensor is used to trigger the camera the moment the wheel passes in front of the camera. In the case of the terminal device 30, a video may be captured and the image of the wheel may be cut out using image processing such as machine learning, or all of the multiple wheel images obtained from the video may be subjected to processing using machine learning, as described below.

[0030] To detect loose nuts, i.e., misalignment between the inner and outer markings, a first machine learning model is created to detect the position of the nuts from images capturing the entire wheel. Wheel nuts are often recessed relative to the tire, making them difficult to focus on or subject to insufficient light. Therefore, it is desirable to train on out-of-focus and dark images to accommodate such situations. Due to the tire's structure, when a vehicle is in motion, its motion is complex, combining linear and rotational motion. The nuts on the top of the tire move in the vehicle's direction of travel and therefore move at high speed, while the nuts on the bottom move in the opposite direction and therefore move at low speed. It is desirable to prepare training data that takes these factors into account when conducting training. Solutions include, for example, using a camera with a sufficiently fast shutter speed or building a system capable of detection even with slight blur. Depending on the type of wheel, a virtual image of the nut may be captured due to specular reflection from the wheel. Therefore, it is desirable to include a variety of wheel types in the training data to detect only the actual nut portion. It may also be possible to detect rust that may occur when the wheel or nut portion is deteriorated, and indicate the degree of danger based on the level of rust.

[0031] As the second machine learning model, a machine learning model is created that detects the outlines of the nut and bolt, the inner markings, and the outer markings for the image of the nut portion detected and cut out by the first machine learning model. Specifically, for the inner markings and the outer markings, point clouds of each marking are detected, but as with the nuts and bolts, the outlines of each marking may also be detected. The training data may include image material of the nut in a loose state, image material in which image editing is performed to recreate a state in which the markings are misaligned, or both. While markings are generally drawn as straight lines, markings are often made as dots. Therefore, it is desirable that the training data include not only image material for line markings but also image material for dot markings. It is desirable that the training data include image material for markings of various shapes, not limited to lines and dots.

[0032] In this embodiment, the first machine learning model and the second machine learning model are separate, but the first machine learning model and the second machine learning model may be created as a single machine learning model. That is, a machine learning model may be employed that determines whether a nut is loose directly from an image of the entire wheel.

[0033] The condition determination device 20 acquires an image of the entire wheel from a stationary camera or the terminal device 30. Using a first machine learning model, the condition determination device 20 detects the position of the nut from the acquired image and cuts out an image of the nut portion. Using a second machine learning model, the condition determination device 20 detects the outlines of the nut and bolt, the inner markings, and the outer markings from the cut-out image of the nut portion. Using a deviation determination program, the condition determination device 20 calculates the angle between the inner marking and the outer marking from the coordinate data of the detected outline and markings, and detects deviations in the markings to determine whether the nut is loose. The condition determination device 20 outputs the determination result to the terminal device 30. When the determination result is input from the condition determination device 20, the terminal device 30 displays the determination result on a screen for the user.

[0034] The camera may be equipped with a light or flash to allow for nighttime use, or may be an infrared camera, in which case the markings employed will be of a type that shows up in infrared images.

[0035] The configuration of a state determination device 20 according to this embodiment will be described with reference to FIG.

[0036] The state determination device 20 includes a control unit 21, a storage unit 22, and a communication unit 23.

[0037] The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. An example of the programmable circuit is an FPGA. "FPGA" is an abbreviation for field-programmable gate array. An example of the dedicated circuit is an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 21 executes processing related to the operation of the state determination device 20 while controlling each part of the state determination device 20.

[0038] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM, a ROM, or a flash memory. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. Flash memory is, for example, an SSD. "SSD" is an abbreviation for solid-state drive. Magnetic memory is, for example, an HDD. "HDD" is an abbreviation for hard disk drive. The storage unit 22 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores data used in the operation of the state determination device 20 and data obtained by the operation of the state determination device 20.

[0039] The communication unit 23 includes at least one communication module. The communication module is, for example, a module that complies with a wired LAN communication standard such as Ethernet (registered trademark) or a wireless LAN communication standard such as IEEE802.11. "IEEE" is an abbreviation for Institute of Electrical and Electronics Engineers. The communication unit 23 communicates with a stationary camera or a terminal device 30 via the network 40. The communication unit 23 receives data used in the operation of the state determination device 20 and transmits data obtained by the operation of the state determination device 20.

[0040] The functions of the state determination device 20 are realized by executing a program according to this embodiment on a processor serving as the control unit 21. That is, the functions of the state determination device 20 are realized by software. The program causes a computer to execute the operations of the state determination device 20, thereby causing the computer to function as the state determination device 20. That is, the computer functions as the state determination device 20 by executing the operations of the state determination device 20 in accordance with the program.

[0041] The program can be stored on a non-transitory computer-readable medium. Examples of non-transitory computer-readable media include flash memory, magnetic recording devices, optical disks, magneto-optical recording media, and ROMs. The program can be distributed by selling, transferring, or lending portable media such as SD cards, DVDs, or CD-ROMs that store the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program can also be distributed by storing it in the storage of a server and transferring it from the server to another computer. The program can also be provided as a program product.

[0042] A computer temporarily stores a program stored on a portable medium or transferred from a server in its main storage device. The computer then reads the program stored in the main storage device using a processor and executes processing in accordance with the read program. The computer may also read the program directly from a portable medium and execute processing in accordance with the program. The computer may also execute processing in accordance with the received program each time a program is transferred from a server to the computer. Processing may also be executed through a so-called ASP-type service that achieves its function simply by issuing an execution command and obtaining the results, without transferring the program from the server to the computer. "ASP" is an abbreviation for application service provider. Programs include information used for processing by a computer that is equivalent to a program. For example, data that does not directly instruct a computer but has properties that define computer processing falls under the category of "equivalent to a program."

[0043] Some or all of the functions of the state determination device 20 may be realized by a programmable circuit or a dedicated circuit as the control unit 21. In other words, some or all of the functions of the state determination device 20 may be realized by hardware.

[0044] The operation of the state determination device 20 according to this embodiment will be described with reference to Fig. 2. The operation described below corresponds to the state determination method according to this embodiment. That is, the state determination method according to this embodiment includes steps S1 to S5 shown in Fig. 2.

[0045] In S1, the control unit 21 acquires the entire wheel image 11 as shown in Fig. 3. Specifically, the control unit 21 receives the entire wheel image 11 from a stationary camera or a terminal device 30 via the communication unit 23. The stationary camera or the terminal device 30 functions as an imaging device that obtains a photographed image 12. The entire wheel image 11 includes an image of each nut portion as the photographed image 12. In the example shown in Fig. 3, the entire wheel image 11 includes a photographed image 12 for each of the six nut portions.

[0046] In S2, the control unit 21 uses the first machine learning model to detect each nut portion from the entire wheel image 11 acquired in S1 and cut out an image of each nut portion. That is, the control unit 21 acquires a captured image 12. The captured image 12 is an image including a first member with a first marking and a second member with a second marking corresponding to the first marking, which are fastening members for fixing the wheel to the vehicle body. In this embodiment, the first member is a nut and the second member is a bolt, but the first member may be a nut or a bolt and the second member may be a member other than a fastening member, such as a member outside the nut.

[0047] In S3, the control unit 21 uses the second machine learning model to detect the contours E1 and E2 of the nuts and bolts, the outer marking M1, and the inner marking M2 from the captured image 12 acquired in S2, as shown in Figures 4 and 5. That is, the control unit 21 detects the contour of the first member, the contour of the second member, the first marking, and the second marking in the captured image 12 acquired in S2. For the outer marking M1, the control unit 21 specifically detects a coordinate group of the marking M1. For the inner marking M2, the control unit 21 specifically detects a coordinate group of the marking M2.

[0048] In S4, the control unit 21 uses a deviation determination program to calculate the angle A between the outer marking M1 and the inner marking M2 from the coordinate data of the contours E1, E2 and the markings M1, M2 detected in S3, as shown in FIGS. 4 to 6. The control unit 21 then determines whether the nut is loose by detecting the deviation between the markings M1 and M2. Specifically, as shown in FIG. 4, the control unit 21 identifies the center position C1 of the contour E1 detected in S3 as the center position of the nut and calculates a first line L1 passing through the center position C1 and the marking M1 on the nut. As shown in FIG. 5, the control unit 21 identifies the center position C2 of the contour E2 detected in S3 as the center position of the bolt and calculates a second line L2 passing through the center position C2 and the marking M2 on the bolt. That is, the control unit 21 identifies the center position of the first member based on the contour of the first member detected in S3. The control unit 21 identifies the center position of the second member based on the contour of the second member detected in S3. The control unit 21 calculates a first line L1 as a line passing through the center position of the first member and the first marking in the captured image 12 acquired in S2, and a second line L2 as a line passing through the center position of the second member and the second marking. As shown in FIG. 6, the control unit 21 calculates an intersection angle A between the first line L1 and the second line L2 in the captured image 12. The control unit 21 determines whether the nut is loose based on whether the calculated angle A exceeds a threshold value. That is, the control unit 21 determines the fastening state of the first member based on the calculated first line L1 and second line L2. The threshold value is, for example, 25°, but can be changed as appropriate.

[0049] A specific method for calculating the first line L1 may be straight-line fitting, i.e., a straight-line approximation method, but other methods may also be used. For example, the control unit 21 may calculate, as the first line L1, a line passing through the center position C1 and, among the coordinate group of the marking M1 affixed to the nut, the coordinate closest to the inner circumference of the nut or the outer circumference of the bolt. Alternatively, the control unit 21 may calculate, as the first line L1, a line passing through the center position C1 and a center of gravity calculated for each coordinate of the marking M1 affixed to the nut by increasing the weighting for coordinates closer to the inner circumference of the nut or the outer circumference of the bolt. That is, the control unit 21 may weight each coordinate included in the coordinate group of the first marking in the captured image 12 according to its relative position, and calculate, as the first line L1, a line passing through the center position of the first member in the captured image 12 and the average coordinate of the coordinate group of the first marking, where each coordinate is weighted.

[0050] The specific method for calculating the second line L2 is similar to the method for calculating the first line L1, and may be a straight-line fitting method, i.e., a straight-line approximation method. However, other methods may also be used. For example, the control unit 21 may calculate, as the second line L2, a line passing through the center position C2 and the coordinates of the marking M2 on the bolt that are closest to the inner circumference of the nut or the outer circumference of the bolt. Alternatively, the control unit 21 may calculate, as the second line L2, a line passing through the center position C2 and the center of gravity calculated for each coordinate of the marking M2 on the bolt by increasing the weighting for coordinates closer to the inner circumference of the nut or the outer circumference of the bolt. That is, the control unit 21 may weight each coordinate included in the coordinate group of the second markings in the captured image 12 according to their relative positions, and calculate, as the second line L2, a line passing through the center position of the second member in the captured image 12 and the average coordinate of the weighted coordinate group of the second markings. As a modified example of this embodiment, the control unit 21 may calculate a line passing through the center position C1 and the marking M2 on the bolt as the second line L2. That is, the control unit 21 may calculate a line passing through the center position of the first member and the second marking as the second line L2.

[0051] In S5, the control unit 21 outputs the result of the determination made in S4. Specifically, the control unit 21 transmits the result of the determination to the terminal device 30 via the communication unit 23. Upon receiving the result of the determination from the state determination device 20, the terminal device 30 displays the result of the determination on a display for the user. Alternatively, the terminal device 30 may output the result of the determination as audio from a speaker for the user. The terminal device 30 functions as an output device that outputs the result of the determination.

[0052] According to the above-described operation, it is possible to reduce the labor and improve the accuracy of inspection of fastening members attached to tire wheels of vehicles.

[0053] As shown in FIG. 7, when the captured image 12 is captured obliquely, the center position of the nut and the center position of the bolt are misaligned. In this embodiment, by calculating a line passing through the center position of the nut for the outer marking M1 and a line passing through the center position of the bolt for the inner marking M2, it is possible to detect the misalignment between the markings M1 and M2 even if the center positions of the nut and the bolt are misaligned. However, if the markings M1 and M2 are misaligned, correction is required to accurately determine the degree of misalignment between the markings M1 and M2. Therefore, in S4, for example, as shown in FIG. 8, the control unit 21 may fit the nut contour E1 to an ellipse F1 to calculate the major and minor axes, and then convert the ellipse F1 back to a circle F1A to correct the angle A between the first line L1 and the second line L2 to the angle AA between the first line L1A and the second line L2A. That is, when calculating the intersection angle between the first line L1 and the second line L2, the control unit 21 may correct the calculated value of the intersection angle in accordance with the shape of the contour E1 detected in S3. Alternatively, the control unit 21 may fit the bolt contour E2 to an ellipse to calculate the major axis and minor axis, and then convert the ellipse back into a circle to correct the angle A between the first line L1 and the second line L2. That is, when calculating the intersection angle between the first line L1 and the second line L2, the control unit 21 may correct the calculated value of the intersection angle in accordance with the shape of the contour E2 detected in S3. In either case, the control unit 21 determines whether the nut is loose depending on whether the corrected angle AA exceeds a threshold value.

[0054] As shown in Figure 9, if a letter is engraved on the head of the bolt, in S4, the control unit 21 may specify the center position of the letter as the center position of the bolt, instead of specifying the center position C2 of the outline E2 detected in S3 as the center position of the bolt. The same applies to cases where a mark other than a letter is engraved on the head of the bolt. As in the example shown in Figure 9, if the letter "R" is engraved on the head of the bolt, it is possible to determine the center of the bolt from this letter using machine learning.

[0055] Nuts and bolts may have new markings of different colors applied without erasing the previous markings. Therefore, when there are multiple color markings, it is desirable to determine the fastening state for each color marking and, if there is no abnormality in the fastening state for at least one color, determine that the nut is not loose. That is, when multiple color markings are applied to the first and second members, the control unit 21 may detect markings of the same color as each other as the first and second markings. In such an example, each time a marking is detected, steps S3 and S4 are executed for the detected marking. If it is determined in S4 that there is no abnormality in the fastening state, step S5 is executed, and the user is notified that there is no abnormality. If it is determined that there is an abnormality in the fastening state for all color markings, step S5 is executed, and the user is notified that there is an abnormality.

[0056] The present disclosure is not limited to the above-described embodiments. For example, two or more blocks shown in the block diagrams may be integrated, or one block may be divided. Two or more steps shown in the flowcharts may be executed in parallel or in a different order, instead of being executed in chronological order as described, depending on the processing capabilities of the device executing each step, or as needed. Other modifications are possible within the scope of the present disclosure.

[0057] [Contribution to the United Nations-led Sustainable Development Goals (SDGs)] The SDGs have been proposed to realize a sustainable society. It is believed that an embodiment of the present disclosure can be a technology that contributes to "No. 9 - Build infrastructure for industry and technological innovation." [Explanation of symbols]

[0058] 10 Systems 11 Full image of the wheel 12 Captured images 20 Status determination device 21 Control section 22 Memory section 23 Communications Department 30 Terminal Equipment 40 Network

Claims

1. acquiring a photographed image including a first member having a first marking, which is a fastening member for fixing a wheel to a vehicle body, and a second member having a second marking corresponding to the first marking; calculating a first straight line passing through a center position of the first member and the first marking in the acquired photographed image, and a second straight line passing through the center position of the first member or the center position of the second member and the second marking; The fastening state of the first member is determined based on the calculated first straight line and the calculated second straight line. A state determination device including a control unit.

2. The control unit calculating an intersection angle between the first straight line and the second straight line in the captured image; The state determination device according to claim 1 , wherein the fastening state is determined to be whether the first member is loosened or not depending on whether the calculated intersection angle exceeds a threshold value.

3. The control unit As the second straight line, a straight line passing through the center position of the second member and the second marking in the photographed image is calculated; Detecting an outline of the first member in the captured image; The state determination device according to claim 2 , wherein when the intersection angle is calculated, the calculated value of the intersection angle is corrected in accordance with the shape of the detected contour.

4. The control unit As the second straight line, a straight line passing through the center position of the second member and the second marking in the photographed image is calculated; Detecting an outline of the second member in the captured image; The state determination device according to claim 2 , wherein when the intersection angle is calculated, the calculated value of the intersection angle is corrected in accordance with the shape of the detected contour.

5. The control unit Detecting an outline of the first member in the captured image; The state determination device according to claim 1 , wherein the center position of the first member is identified based on the detected contour.

6. The control unit weighting each coordinate included in the coordinate group of the first marking in the captured image according to a relative position; The state determination device according to claim 1 , wherein the first straight line is calculated as a straight line passing through the center position of the first member in the captured image and an average coordinate of the group of coordinates of the first marking, each of which is weighted.

7. 2. The state determination device according to claim 1, wherein, when the first member and the second member each have markings of multiple colors, the control unit detects markings of the same color as each other as the first marking and the second marking.

8. The state determination device according to claim 1 , wherein the first member is a nut or a bolt.

9. The state determination device according to claim 1 , wherein the first member is a nut and the second member is a bolt.

10. The state determination device according to any one of claims 1 to 9, an imaging device for obtaining the photographed image; A system comprising:

11. A computer acquires a photographed image including a first member having a first marking, which is a fastening member for fixing a wheel to a vehicle body, and a second member having a second marking corresponding to the first marking; the computer calculates a first line passing through a center position of the first member and the first marking in the acquired photographed image, and a second line passing through the center position of the first member or the center position of the second member and the second marking; the computer determines a fastening state of the first member based on the calculated first straight line and the calculated second straight line; A state determination method including:

12. Acquiring a photographed image including a first member having a first marking and a second member having a second marking corresponding to the first marking, the first member being a fastening member for fixing a wheel to a vehicle body; calculating a first straight line passing through a center position of the first member and the first marking in the acquired photographed image, and a second straight line passing through the center position of the first member or the center position of the second member and the second marking; determining a fastening state of the first member based on the calculated first straight line and the calculated second straight line; A program that causes a computer to perform operations including:

Images