Bolt assembly detection method, device, computing equipment and storage medium based on visual large model

Abstract

The present invention relates to the field of artificial intelligence technology, and more particularly to a method, apparatus, computing device, and storage medium for bolt assembly detection based on a visual large model, wherein the method comprises: acquiring a target image through a wearable device, and identifying visible bolt targets contained in the target image through a visual large model; determining a reference bolt target among the visible bolt targets, and establishing bolt codes corresponding to all visible bolt targets based on the reference bolt target; determining a predicted assembly sequence based on the bolt codes, and determining the actual assembly sequence of all visible bolt targets; generating bolt assembly information based on the predicted assembly sequence and the actual assembly sequence, and constructing an assembly sequence alarm instruction based on the bolt assembly information. The present invention adapts to the installation requirements of workpieces of different sizes in clockwise and counterclockwise order, ensures the standardization of assembly operations without affecting normal operations, improves work efficiency, and reduces equipment investment and management difficulty.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a bolt assembly detection method, device, computing equipment and storage medium based on a visual large model. Background Art

[0002] Ring bolt group connections are commonly used to connect two parts in the form of tubes or barrels. In actual use, to ensure the tight and reliable connection between the two tubular parts and avoid leakage of internal gas and liquid, most of them use pairs of bolts that are evenly distributed for connection. Therefore, the reliability of the bolt connection has a direct impact on the assemblability of the entire product, and the quality of the bolt connection is mainly affected by the bolt assembly process, among which the tightening sequence of the bolts is a key process step. Currently, in the mechanical field at home and abroad, many of these connections are assembled manually, and the assembly sequence of the ring bolt group is also completed by the operator. Due to the large number of bolt groups, the tightening sequence is currently determined by marking the bolt holes on the parts, which is time-consuming, labor-intensive, and prone to errors. Therefore, it is very necessary to add a monitoring system to guide workers during installation and identify incorrect assembly sequences. Computer vision technology can identify and process assemblies in real time and make logical judgments, and is a necessary means to achieve intelligence.

[0003] Current technology primarily relies on fixing cameras above mechanical workpieces to capture an overall view. However, this approach has several significant drawbacks. First, hardware costs are relatively high. Because different sizes of mechanical workpieces require dedicated fixed cameras, each size must be equipped with a separate set of storage platforms and camera equipment, which undoubtedly increases the company's equipment investment. Second, this method lacks flexibility, especially for workpieces that are large and difficult to move. Using this fixed camera approach requires a separate camera for each workpiece, significantly increasing resource consumption and management costs.

[0004] Furthermore, existing methods are inherently inadequate when dealing with moving camera scenes. Because the shape of a bolt changes with viewing angle, existing methods rely solely on the change in pixels between the front and back to determine whether a worker has tightened the bolt. This approach is significantly less accurate when directly applied to moving camera scenes, increasing the risk of misjudgment and impacting overall monitoring effectiveness. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method for bolt assembly detection based on a large visual model to address the technical deficiencies in the prior art. Embodiments of the present invention also provide a bolt assembly detection device based on a large visual model, a computing device, and a computer-readable storage medium.

[0006] According to a first aspect of an embodiment of the present invention, a method for detecting bolt assembly based on a visual large model is provided, comprising:

[0007] Capturing a target image through a wearable device, and identifying visible bolt targets contained in the target image through a visual large model;

[0008] Determine a reference bolt target among the visible bolt targets, and establish bolt codes corresponding to all the visible bolt targets based on the reference bolt target;

[0009] Determining a predicted assembly sequence according to the bolt codes, and determining an actual assembly sequence of all the visible bolt targets;

[0010] Bolt assembly information is generated according to the predicted assembly sequence and the actual assembly sequence, and an assembly sequence warning instruction is constructed according to the bolt assembly information.

[0011] Optionally, identifying visible bolt targets contained in the target image by using a visual large model includes:

[0012] Based on the pre-trained visual large model, the visible bolt target contained in the target image is identified, wherein the visible bolt target includes a tightened bolt state and an untightened bolt state.

[0013] Optionally, determining a reference bolt target among the visible bolt targets includes:

[0014] When all the visible bolt objects in the object image are in an undightened state, the first visible bolt object that changes from an undightened state to a tightened state is determined as the reference bolt object.

[0015] Optionally, the step of establishing bolt codes corresponding to all visible bolt targets based on the reference bolt target includes:

[0016] Define the bolt code corresponding to the benchmark bolt target as 0;

[0017] The bolt codes of the remaining visible bolt targets are defined from near to far according to the distances between the bolt targets and the reference bolt targets, wherein the value range of the bolt codes is an integer.

[0018] Optionally, after establishing the bolt codes corresponding to all the visible bolt targets based on the reference bolt target, the method further includes:

[0019] Based on the image acquisition result of the wearable device, updating the target image;

[0020] identifying a newly added bolt target in the target image;

[0021] Determine the visible bolt target adjacent to the newly added bolt target as a reference bolt target;

[0022] Based on the bolt code corresponding to the reference bolt target, a bolt code corresponding to the newly added bolt target is defined.

[0023] Optionally, when the newly added bolt target is in a tightened state, defining the bolt code corresponding to the newly added bolt target based on the bolt code corresponding to the reference bolt target includes:

[0024] If the number of the reference bolt targets is 1, the bolt code corresponding to the newly added bolt target is defined according to the positive and negative values ​​of the bolt codes corresponding to the reference bolt targets and the positional relationship between the reference bolt targets and the newly added bolt targets;

[0025] If the number of the reference bolt targets is 2, the interpolation of the bolt codes corresponding to the two reference bolt targets is calculated, and the bolt code corresponding to the newly added bolt target is defined according to the interpolation calculation result.

[0026] Optionally, when the newly added bolt target is in an untightened state, defining the bolt code corresponding to the newly added bolt target based on the bolt code corresponding to the reference bolt target includes:

[0027] When the reference bolt target has a corresponding non-zero bolt code, the bolt code corresponding to the newly added bolt target is defined according to the non-zero bolt code corresponding to the reference bolt target;

[0028] When the reference bolt target does not have a corresponding non-zero bolt code, the bolt code corresponding to the newly added bolt target is defined according to the bolt code of 0 corresponding to the reference bolt target.

[0029] According to a second aspect of an embodiment of the present invention, a bolt assembly detection device based on a visual large model is provided, comprising:

[0030] a recognition module configured to collect a target image through a wearable device and recognize a visible bolt target contained in the target image through a visual macro model;

[0031] an encoding module configured to determine a reference bolt target among the visible bolt targets, and establish bolt codes corresponding to all the visible bolt targets based on the reference bolt target;

[0032] a sequence module configured to determine a predicted assembly sequence based on the bolt codes and determine an actual assembly sequence of all the visible bolt targets;

[0033] The alarm module is configured to generate bolt assembly information according to the predicted assembly sequence and the actual assembly sequence, and construct an assembly sequence alarm instruction according to the bolt assembly information.

[0034] According to a third aspect of an embodiment of the present invention, there is provided a computing device, including:

[0035] memory and processor;

[0036] The memory is used to store computer-executable instructions, and the processor implements the steps of the bolt assembly detection method based on the visual large model when executing the computer-executable instructions.

[0037] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, which stores computer-executable instructions. When the instructions are executed by a processor, the steps of the bolt assembly detection method based on the visual large model are implemented.

[0038] According to a fifth aspect of an embodiment of the present invention, a chip is provided, which stores a computer program. When the computer program is executed by the chip, the steps of the bolt assembly detection method based on the visual large model are implemented.

[0039] The present invention provides a bolt assembly detection method, apparatus, computing device, and storage medium based on a visual large model, wherein the method captures a target image through a wearable device and identifies visible bolt targets contained in the target image through a visual large model; determines a reference bolt target among the visible bolt targets, and establishes bolt codes corresponding to all the visible bolt targets based on the reference bolt targets; determines a predicted assembly sequence based on the bolt codes, and determines the actual assembly sequence of all the visible bolt targets; generates bolt assembly information based on the predicted assembly sequence and the actual assembly sequence, and constructs an assembly sequence alarm instruction based on the bolt assembly information. The present invention adapts to the installation requirements of workpieces of different sizes in clockwise and counterclockwise order, ensures the standardization of assembly operations without affecting normal operations, improves work efficiency, and reduces equipment investment and management difficulty. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a flow chart of a bolt assembly detection method based on a visual large model provided by one embodiment of the present invention;

[0042] Figure 2 This is a processing flow chart of a bolt assembly detection method based on a visual large model provided by one embodiment of the present invention;

[0043] Figure 3 Schematic diagram of a bolt in a bolt assembly detection method based on a visual large model provided by an embodiment of the present invention;

[0044] Figure 4 1 is a schematic structural diagram of a bolt assembly detection device based on a visual large model provided by an embodiment of the present invention;

[0045] Figure 5 This is a structural block diagram of a computing device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following description sets forth numerous specific details to facilitate a thorough understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific implementations disclosed below.

[0047] The terms used in one or more embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of the present invention. The singular forms "a", "the" and "the" used in one or more embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present invention refers to and includes any or all possible combinations of one or more associated listed items.

[0048] The present invention provides a method for detecting bolt assembly based on a large visual model. The present invention also relates to a device for detecting bolt assembly based on a large visual model, a computing device, and a computer-readable storage medium, which are described in detail in the following embodiments.

[0049] Figure 1 A flowchart of a bolt assembly detection method based on a visual large model according to an embodiment of the present invention is shown, which specifically includes the following steps:

[0050] Step S102: collecting a target image through a wearable device, and identifying visible bolt targets contained in the target image through a visual macro model;

[0051] Step S104: determining a reference bolt target among the visible bolt targets, and establishing bolt codes corresponding to all the visible bolt targets based on the reference bolt target;

[0052] Step S106: determining a predicted assembly sequence according to the bolt codes, and determining an actual assembly sequence of all the visible bolt targets;

[0053] Step S108: generating bolt assembly information according to the predicted assembly sequence and the actual assembly sequence, and constructing an assembly sequence warning instruction according to the bolt assembly information.

[0054] The large visual model is based on deep learning technology, specifically the Transformer architecture, and is used to process and analyze image data. Through training on massive amounts of data, it can automatically extract feature information from images, enabling complex tasks such as image classification, target detection, and image segmentation. The wearable device is an image acquisition device worn by the operator. The image data collected by the image acquisition device is transmitted to the image processing device for processing to achieve bolt assembly inspection. The image processing device is integrated into the wearable device, or the image processing device is an independent hardware device that receives image data sent by the data transmission device in the wearable device. The target image is acquired during the operation of the operator wearing the wearable device, and includes images that require bolt assembly inspection.

[0055] Based on this, during the operation, the operator collects the target image through the wearable device, and the image processing equipment identifies the visible bolt targets according to the target image; then the first assembled visible bolt target is located as the reference bolt target, and all visible bolt targets are encoded according to the located reference bolt target to obtain bolt codes that correspond one-to-one to the visible bolt targets.

[0056] Furthermore, the predicted assembly sequence is the order in which the visible bolt targets are assembled when the bolts are assembled in a prescribed clockwise or counterclockwise direction, as predicted by the image processing device. The actual assembly sequence is the order in which the visible bolt targets are actually assembled when the operator assembles the bolts. After obtaining the bolt codes, the predicted assembly sequence is determined based on the bolt codes, and the operator's actual assembly sequence is monitored via a wearable device. Bolt assembly information is generated by comparing the predicted assembly sequence with the actual assembly sequence. If the bolt assembly information shows that the predicted assembly sequence is the same as the actual assembly sequence, indicating that the assembly is correct, an assembly sequence alarm instruction will not be sent to the preset alarm device. If the bolt assembly information shows that the predicted assembly sequence is different from the actual assembly sequence, indicating that the operator's assembly sequence does not meet the requirements, an assembly sequence alarm instruction will be sent to the alarm device for alarm. The alarm method includes light, sound, text, etc. The specific alarm method is determined by actual usage requirements and is not limited in this embodiment.

[0057] Furthermore, in the above step S102, the process of identifying the visible bolt targets contained in the target image by using the visual large model is specifically implemented as follows in this embodiment:

[0058] Based on the pre-trained visual large model, the visible bolt target contained in the target image is identified, wherein the visible bolt target includes a tightened bolt state and an untightened bolt state.

[0059] The bolted state indicates that the stud of the visible bolt target has a nut, while the unbolted state indicates that the stud of the visible bolt target does not have a nut. Figure 2 As shown in the processing flow chart of a bolt assembly detection method based on a visual large model, a wearable camera device acquires an image, then performs instance segmentation to obtain bolts. The wearable camera device is the wearable device, the image is the target image, and the instance segmentation is implemented by a visual large model using a segmentation algorithm. The obtained bolts are visible bolt targets.

[0060] Furthermore, in the above step S104, the process of determining the reference bolt target among the visible bolt targets is specifically implemented as follows in this embodiment:

[0061] When all the visible bolt objects in the object image are in an undightened state, the first visible bolt object that changes from an undightened state to a tightened state is determined as the reference bolt object.

[0062] Among them, for the positioning process of the first assembled visible bolt target, that is, the reference bolt target, it is necessary to determine whether all visible bolt targets are in the untightened state, and the first visible bolt target that changes from the untightened state to the tightened state is defined as the reference bolt target.

[0063] Based on this, there is also a situation where the operator leaves in the middle of assembling bolts on a workpiece. At this time, the first frame acquired contains a target image of a visible bolt target, in which the visible bolt target exists in both a tightened bolt state and an untightened bolt state. Since the operator's leaving in the middle of the process is an illegal operation, the operator's subsequent bolt assembly based on the memory before leaving cannot guarantee the correctness of the assembly sequence. At this time, the operator must remove all the bolts and reassemble them. If the operator does not remove them and chooses to continue assembling the bolts, an assembly sequence alarm instruction is generated, instructing the alarm device to alarm.

[0064] Furthermore, the above step S104 is a process of establishing bolt codes corresponding to all visible bolt targets based on the reference bolt target. In this embodiment, the specific implementation is as follows:

[0065] The bolt code corresponding to the reference bolt target is defined as 0; and the bolt codes of the remaining visible bolt targets are defined from near to far according to the distance between the reference bolt target and the bolt target, wherein the value range of the bolt code is an integer.

[0066] Among them, such as Figure 2 The process flow diagram of a bolt assembly inspection method based on a large visual model is shown. When the first tightened bolt appears, its ID is initialized to 0. The distances between bolt 0 and the remaining bolts in the field of view are then calculated and sorted by distance. The remaining bolts are manually assigned IDs based on their distances. It should be noted that the first tightened bolt and bolt 0 are the reference bolt targets, the IDs are the bolt codes, and the remaining bolts in the field of view are the visible bolt targets besides the reference bolt target.

[0067] Based on this, in actual usage scenarios, the visual large model of the segmentation algorithm is used to process the target image. The minimum enclosing rectangle of each bolt is used as the initial frame of the multi-target tracking algorithm. The bolt code of the first visible bolt target that becomes a tightened bolt is defined as 0. The other visible bolt targets are sorted according to the distance from the reference bolt. The farther the distance, the larger the bolt code. For example, the bolt code on the left side of the reference bolt target is assigned a negative value, which is -1, -2, -3,... from near to far, and all the bolt codes corresponding to the counterclockwise order are obtained. Correspondingly, the bolt code on the right side is assigned a positive value, which is 1, 2, 3,... from near to far, and all the bolt codes corresponding to the clockwise order are obtained. This adapts to the movement of people in different directions and achieves compatibility with both clockwise and counterclockwise possible operation sequences.

[0068] It should be noted that the above relationship of negative bolt codes for counterclockwise pins and positive bolt codes for clockwise pins can also be positive for counterclockwise pins and negative for clockwise pins. The specific correspondence is determined by the actual usage scenario and is not limited in this embodiment. To ensure that the initial frame of the multi-target tracking algorithm does not lose the visible bolt targets, the operator's hand target is synchronously segmented, and multiple visible bolt targets are tracked based on the distance between the hand target and the visible bolt targets.

[0069] Furthermore, since the operator moves a large workpiece during bolt assembly, the position and number of visible bolt targets in the target image captured by the wearable device will change. Some new bolts will appear in the target image, while some old bolts will move out of the target image. At this time, the bolt code needs to be updated. The updating process is specifically implemented as follows in this embodiment:

[0070] Based on the image acquisition result of the wearable device, the target image is updated; a new bolt target in the target image is identified; the visible bolt target adjacent to the new bolt target is determined as a reference bolt target; and based on the bolt code corresponding to the reference bolt target, the bolt code corresponding to the new bolt target is defined.

[0071] New bolt targets are visible bolt targets that did not exist in the target image before the update but are present in the target image after the update. In this case, there are two scenarios for new bolt targets: the first is when the operator's hand, arm, or wrench obscures the visible bolt target during bolt assembly; the second is when the operator's constant movement during bolt assembly causes an unassembled bolt to enter the target image.

[0072] Based on this, Figure 2 The process flow diagram of a bolt assembly inspection method based on a large visual model is shown. A person moves with a camera, capturing new image frames. This wearable device performs real-time image acquisition and continuously updates the target image based on the captured frames. Then, using a multi-target tracking algorithm, if a new bolt appears, the IDs of the two currently visible bolt boundaries are obtained and a new bolt ID is assigned based on the distance. The new bolt is the newly added bolt target. The IDs of the two currently visible bolt boundaries are the bolt codes corresponding to the reference bolt target, and the new bolt ID is the bolt code corresponding to the newly added bolt target.

[0073] It should be noted that the method for determining whether the two reference bolt targets are on either side of the newly added bolt target is to calculate the center points of the minimum bounding rectangle of each of the three bolts. Let these three points be point A (x1, y1), point B (x2, y2), and point C (x, y), where point C is the center point of the newly added bolt target. Construct vector CB = (x2-x, y2-y) and vector CA = (x1-x, y1-y), and calculate the angle between these vectors. If the angle is greater than 90°, the two reference bolt targets are considered to be on opposite sides of the newly added bolt target; otherwise, they are considered to be on the same side.

[0074] Furthermore, if the first of the above two situations is the case, then based on the bolt code corresponding to the reference bolt target, the process of defining the bolt code corresponding to the newly added bolt target is specifically implemented as follows in this embodiment:

[0075] If the number of reference bolt targets is 1, the bolt code corresponding to the newly added bolt target is defined based on the positive and negative values ​​of the bolt code corresponding to the reference bolt target and the positional relationship between the reference bolt target and the newly added bolt target; if the number of reference bolt targets is 2, the interpolation of the bolt codes corresponding to the two reference bolt targets is calculated, and the bolt code corresponding to the newly added bolt target is defined based on the interpolation calculation result.

[0076] Among them, when the newly added bolt target is a visible bolt target that is blocked, then the newly added bolt target should be a bolt that has been successfully assembled, so it is in the tightened bolt state.

[0077] Based on this, if the number of reference bolt targets is 1, in order to restore the bolt code corresponding to the newly added bolt target, first determine the positive and negative values ​​of the bolt code corresponding to the reference bolt target. If it is a positive value, then the bolt code corresponding to the newly added bolt target is also a positive value. Conversely, if the bolt code of the reference bolt target is a negative value, then the bolt code of the newly added bolt target is also a negative value. Then, based on the positive and negative value judgment results, determine whether the bolt assembly is carried out clockwise or counterclockwise.

[0078] If it is clockwise, and the bolt code corresponding to the reference bolt target is a positive value, then the corresponding bolt code of the newly added bolt target immediately to the right of the reference bolt target is the bolt code of the reference bolt target plus 1; if it is clockwise, and the bolt code corresponding to the reference bolt target is a positive value, then the corresponding bolt code of the newly added bolt target immediately to the left of the reference bolt target is the bolt code of the reference bolt target plus -1; if it is clockwise, and the bolt code corresponding to the reference bolt target is a negative value, then the corresponding bolt code of the newly added bolt target immediately to the right of the reference bolt target is the bolt code of the reference bolt target plus -1; if it is clockwise, and the bolt code corresponding to the reference bolt target is a negative value, then the corresponding bolt code of the newly added bolt target immediately to the left of the reference bolt target is the bolt code of the reference bolt target plus 1; If the rotation is counterclockwise and the bolt code corresponding to the reference bolt target is positive, the bolt code corresponding to the newly added bolt target immediately to the right of the reference bolt target is the bolt code of the reference bolt target plus -1. If the rotation is counterclockwise and the bolt code corresponding to the reference bolt target is positive, the bolt code corresponding to the newly added bolt target immediately to the left of the reference bolt target is the bolt code of the reference bolt target plus 1. If the rotation is counterclockwise and the bolt code corresponding to the reference bolt target is negative, the bolt code corresponding to the newly added bolt target immediately to the right of the reference bolt target is the bolt code of the reference bolt target plus 1. If the rotation is counterclockwise and the bolt code corresponding to the reference bolt target is negative, the bolt code corresponding to the newly added bolt target immediately to the left of the reference bolt target is the bolt code of the reference bolt target plus -1. If the number of newly added bolt targets is not 1, the newly added bolt target with the most recently confirmed bolt code can be used as the reference bolt target, and the bolt codes of the newly added bolt targets can be assigned in sequence.

[0079] In addition, if the number of reference bolt targets is 2, such as Figure 2 As shown in the processing flow chart of the bolt assembly detection method based on a visual large model, if the target bolt loses its ID due to occlusion and then reappears, the ID is reassigned by interpolating the IDs of the adjacent bolts on both sides. Specifically, the bolt code of the newly added bolt target is an integer between the bolt codes of the two reference bolt targets. The bolt code of the newly added bolt target can be determined by interpolation. In addition, it should be noted that when the number of newly added bolt targets is not 1, it is necessary to ensure that the arrangement of the numerical values ​​of the newly added bolt target codes determined by interpolation is consistent with the arrangement of the numerical values ​​of the bolt codes of the reference bolt targets.

[0080] Furthermore, in the second case, based on the bolt code corresponding to the reference bolt target, a process of defining a bolt code corresponding to a newly added bolt target is specifically implemented as follows in this embodiment:

[0081] When the reference bolt target has a corresponding non-zero bolt code, the bolt code corresponding to the newly added bolt target is defined according to the non-zero bolt code corresponding to the reference bolt target;

[0082] When the reference bolt target does not have a corresponding non-zero bolt code, the bolt code corresponding to the newly added bolt target is defined according to the bolt code of 0 corresponding to the reference bolt target.

[0083] Among them, since the newly added bolt target is an unassembled bolt, the newly added bolt target is in an untightened bolt state. At this time, there are also two situations. If there is a bolt code that is not 0 in the bolt code corresponding to the reference bolt target, in this case, it is necessary to define the bolt code of the newly added bolt target based on the bolt code of the reference bolt target that is not 0. For the specific definition method, please refer to the above-mentioned method in which the newly added bolt target is in a tightened bolt state and the number of reference bolt targets is 1, which will not be repeated in this embodiment. This avoids the situation where the operator assembles bolts for a week and at the end stage, there are two reference bolt targets on the target image, one of which is the base bolt target. The bolt code of the newly added bolt target is defined according to the bolt code of the base bolt target, resulting in a jump value between all the bolt codes.

[0084] For example, Figure 3 As shown in the bolt schematic diagram of a bolt assembly detection method based on a visual large model, the target image of the previous frame contains visible bolt targets, whose corresponding bolt codes are 2-5. The newly appeared bolt is the newly added bolt target in the target image of the current frame. At this time, the reference bolt target is determined to be the visible bolt target with a bolt code of 5. At this time, the bolt code is judged to be a positive value, so the bolt code 5 is increased by 1, and the obtained value is 6, which is the bolt code corresponding to the newly added bolt target.

[0085] In addition, if the reference bolt target only has a bolt code of 0, then it is necessary to define the bolt code of the newly added bolt target based on the bolt code of the reference bolt target of 0. The specific definition method is described in the above method where the newly added bolt target is in the tightened bolt state and the number of reference bolt targets is 1. This embodiment will not repeat it in detail.

[0086] Afterwards, the execution process of step S106 and step S108 is as follows: Figure 2As shown in the processing flow chart of a bolt assembly detection method based on a visual large model, if there is a new tightened bolt, its ID is recorded and compared with the result of predicting the next tightened bolt ID based on the tightened bolt ID list. If the new tightened bolt ID is consistent with the predicted ID, that is, "the ID is in the predicted ID", then the bolt assembly sequence is correct, and the ID of the new tightened bolt is updated to the tightened bolt ID list. If it is inconsistent, then the bolt assembly sequence is incorrect and an alarm is required.

[0087] Furthermore, the determination of whether all bolts on a workpiece are assembled is completed, e.g. Figure 2 As shown in the figure, if the judgment condition that one circle of bolts has been tightened is met, the task ends; if it is not met, the detection continues. The judgment condition that one circle of bolts has been tightened is that when there is only one visible bolt target in the target image in the untightened state, and the two closest visible bolt targets are both tightened, and these two tightened visible bolt targets are located on both sides of the visible bolt target in the untightened state, the current visible bolt target in the untightened state is determined to be the last untightened bolt. When it changes to the tightened state, the ring bolt group is considered to have been tightened for one circle of bolts, and the task ends.

[0088] Corresponding to the above method embodiment, the present invention also provides an embodiment of a bolt assembly detection device based on a visual large model, Figure 4 FIG1 shows a schematic diagram of a bolt assembly detection device based on a visual large model provided by an embodiment of the present invention. Figure 4 As shown, the device includes:

[0089] The recognition module 402 is configured to collect a target image through a wearable device and recognize a visible bolt target contained in the target image through a visual macro model;

[0090] The encoding module 404 is configured to determine a reference bolt target among the visible bolt targets, and establish bolt codes corresponding to all the visible bolt targets based on the reference bolt target;

[0091] A sequence module 406 is configured to determine a predicted assembly sequence based on the bolt codes and determine an actual assembly sequence of all the visible bolt targets;

[0092] The alarm module 408 is configured to generate bolt assembly information according to the predicted assembly sequence and the actual assembly sequence, and construct an assembly sequence alarm instruction according to the bolt assembly information.

[0093] In an optional embodiment, the identification module 402 is further configured to:

[0094] Based on the pre-trained visual large model, the visible bolt target contained in the target image is identified, wherein the visible bolt target includes a tightened bolt state and an untightened bolt state.

[0095] In an optional embodiment, the encoding module 404 is further configured to:

[0096] When all the visible bolt objects in the target image are in an undightened state, the first visible bolt object that changes from an undightened state to a tightened state is determined as the reference bolt object.

[0097] In an optional embodiment, the encoding module 404 is further configured to:

[0098] The bolt code corresponding to the reference bolt target is defined as 0; and the bolt codes of the remaining visible bolt targets are defined from near to far according to the distance between the reference bolt target and the bolt target, wherein the value range of the bolt code is an integer.

[0099] In an optional embodiment, the bolt assembly detection device based on a visual large model further includes:

[0100] The update module is configured to update the target image based on the image acquisition result of the wearable device; identify a newly added bolt target in the target image; determine the visible bolt target adjacent to the newly added bolt target and in a tightened state as a reference bolt target; and define the bolt code corresponding to the newly added bolt target based on the bolt code corresponding to the reference bolt target.

[0101] In an optional embodiment, the update module is further configured to:

[0102] When the newly added bolt target is in the tightened bolt state, if the number of the reference bolt targets is 1, the bolt code corresponding to the newly added bolt target is defined based on the positive and negative values ​​of the bolt codes corresponding to the reference bolt targets and the positional relationship between the reference bolt targets and the newly added bolt targets; if the number of the reference bolt targets is 2, the interpolation of the bolt codes corresponding to the two reference bolt targets is calculated, and the bolt code corresponding to the newly added bolt target is defined based on the interpolation calculation result.

[0103] In an optional embodiment, the update module is further configured to:

[0104] When the newly added bolt target is in an untightened state and the reference bolt target has a corresponding non-0 bolt code, the bolt code corresponding to the newly added bolt target is defined according to the non-0 bolt code corresponding to the reference bolt target; when the reference bolt target does not have a corresponding non-0 bolt code, the bolt code corresponding to the newly added bolt target is defined according to the bolt code that is 0 corresponding to the reference bolt target.

[0105] The bolt assembly detection device based on a visual macromodel provided by the present invention collects a target image through a wearable device and identifies visible bolt targets contained in the target image using a visual macromodel. It then determines a reference bolt target among the visible bolt targets and, based on the reference bolt target, establishes bolt codes corresponding to all the visible bolt targets. It then determines a predicted assembly sequence based on the bolt codes and the actual assembly sequence for all the visible bolt targets. It then generates bolt assembly information based on the predicted and actual assembly sequences, and constructs an assembly sequence warning instruction based on the bolt assembly information. The present invention accommodates the clockwise and counterclockwise installation requirements for workpieces of varying sizes. Without affecting normal operation, it ensures the standardization of assembly operations, improves work efficiency, and reduces equipment investment and management difficulty.

[0106] The above is a schematic scheme of a bolt assembly detection device based on a visual large model in this embodiment. It should be noted that the technical solution of the bolt assembly detection device based on a visual large model and the technical solution of the bolt assembly detection method based on a visual large model belong to the same concept. For the details not described in detail in the technical solution of the bolt assembly detection device based on a visual large model, please refer to the description of the technical solution of the bolt assembly detection method based on a visual large model. In addition, each component in the embodiment of the device should be understood as a functional module that must be established to implement each step of the program flow or each step of the method, and each functional module is not an actual functional division or separation definition. The device claim defined by such a group of functional modules should be understood as a functional module architecture that mainly implements the solution through the computer program recorded in the specification, and should not be understood as a physical device that mainly implements the solution through hardware.

[0107] Figure 5 1 shows a block diagram of a computing device 500 according to an embodiment of the present invention. Components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0108] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)), whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0109] In one embodiment of the present invention, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 5 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of the present invention. Those skilled in the art may add or replace other components as needed.

[0110] Computing device 500 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 500 can also be a mobile or stationary server.

[0111] The processor 520 is configured to execute computer executable instructions for each step of the visual large model-based bolt assembly detection method.

[0112] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device is based on the same concept as the technical solution of the aforementioned visual large-scale model-based bolt assembly detection method. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the aforementioned visual large-scale model-based bolt assembly detection method.

[0113] An embodiment of the present invention further provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to execute the steps of the bolt assembly detection method based on a visual large model.

[0114] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solution of the aforementioned visual large-scale model-based bolt assembly detection method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the aforementioned visual large-scale model-based bolt assembly detection method.

[0115] An embodiment of the present invention further provides a chip storing a computer program, which implements the steps of the bolt assembly detection method based on the visual large model when executed by the chip.

[0116] The foregoing description describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0117] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium may be appropriately increased or decreased based on the requirements of legislation and patent practice within a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0118] It should be noted that for the aforementioned method embodiments, for ease of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0119] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0120] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. The alternative embodiments do not describe all details in detail, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the teachings of the present invention. These embodiments are selected and described in detail to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A bolt assembly detection method based on a visual large model, characterized in that: include: Capturing a target image through a wearable device, and identifying visible bolt targets contained in the target image through a visual macro model. Specifically, based on the pre-trained visual macro model, identifying the visible bolt targets contained in the target image, wherein the visible bolt targets include a tightened bolt state and an untightened bolt state; Determine a reference bolt target among the visible bolt targets, and establish bolt codes corresponding to all the visible bolt targets based on the reference bolt target. Specifically, when all the visible bolt targets in the target image are in an untightened state, determine the first visible bolt target that changes from an untightened state to a tightened state as the reference bolt target, define the bolt code corresponding to the reference bolt target as 0, and define the bolt codes of the remaining visible bolt targets from near to far according to the distance between them and the reference bolt target, wherein the value range of the bolt code is an integer; Determining a predicted assembly sequence according to the bolt codes, and determining an actual assembly sequence of all the visible bolt targets; Bolt assembly information is generated according to the predicted assembly sequence and the actual assembly sequence, and an assembly sequence warning instruction is constructed according to the bolt assembly information.

2. The method according to claim 1, characterized in that After establishing the bolt codes corresponding to all the visible bolt targets based on the reference bolt target, the method further includes: Based on the image acquisition result of the wearable device, updating the target image; identifying a newly added bolt target in the target image; Determine the visible bolt target adjacent to the newly added bolt target as a reference bolt target; Based on the bolt code corresponding to the reference bolt target, a bolt code corresponding to the newly added bolt target is defined.

3. The method according to claim 2, characterized in that When the newly added bolt target is in a tightened state, defining the bolt code corresponding to the newly added bolt target based on the bolt code corresponding to the reference bolt target includes: If the number of the reference bolt targets is 1, the bolt code corresponding to the newly added bolt target is defined according to the positive and negative values ​​of the bolt codes corresponding to the reference bolt targets and the positional relationship between the reference bolt targets and the newly added bolt targets; If the number of the reference bolt targets is 2, the interpolation of the bolt codes corresponding to the two reference bolt targets is calculated, and the bolt code corresponding to the newly added bolt target is defined according to the interpolation calculation result.

4. The method according to claim 2, characterized in that When the newly added bolt target is in an untightened state, defining the bolt code corresponding to the newly added bolt target based on the bolt code corresponding to the reference bolt target includes: When the reference bolt target has a corresponding non-zero bolt code, the bolt code corresponding to the newly added bolt target is defined according to the non-zero bolt code corresponding to the reference bolt target; When the reference bolt target does not have a corresponding non-zero bolt code, the bolt code corresponding to the newly added bolt target is defined according to the bolt code of 0 corresponding to the reference bolt target.

5. A bolt assembly detection device based on a visual large model, characterized in that: include: a recognition module configured to capture a target image through a wearable device and identify visible bolt targets contained in the target image using a visual macro model. Specifically, based on the pre-trained visual macro model, the recognition module identifies the visible bolt targets contained in the target image, wherein the visible bolt targets include a tightened bolt state and an untightened bolt state. The encoding module is configured to determine a reference bolt target among the visible bolt targets, and establish bolt codes corresponding to all the visible bolt targets based on the reference bolt target. Specifically, when all the visible bolt targets in the target image are in an untightened state, the first visible bolt target that changes from an untightened state to a tightened state is determined as the reference bolt target, and the bolt code corresponding to the reference bolt target is defined as 0. The bolt codes of the remaining visible bolt targets are defined from near to far according to the distance between the visible bolt targets and the reference bolt target, wherein the value range of the bolt code is an integer; a sequence module configured to determine a predicted assembly sequence based on the bolt codes and determine an actual assembly sequence of all the visible bolt targets; The alarm module is configured to generate bolt assembly information according to the predicted assembly sequence and the actual assembly sequence, and construct an assembly sequence alarm instruction according to the bolt assembly information.

6. A computing device, characterized in that include: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the bolt assembly detection method based on visual large models as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by the processor, the steps of the bolt assembly detection method based on the visual large model described in any one of claims 1 to 4 are implemented.

Images