Bolt assembly detection method and device based on visual large model, computing equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a bolt assembly detection method and device based on a visual large model, computing equipment and a storage medium, and the method comprises the steps: collecting a target image through wearable equipment, and recognizing a visible bolt target contained in the target image through the visual large model; determining a reference bolt target in the visible bolt targets, and establishing bolt codes corresponding to all the visible bolt targets based on the reference bolt target; determining a predicted assembly sequence according to the bolt codes, and determining an actual assembly sequence of all visible bolt targets; and generating bolt assembly information according to the predicted assembly sequence and the actual assembly sequence, and constructing an assembly sequence alarm instruction according to the bolt assembly information. According to the invention, clockwise and anticlockwise mounting requirements of workpieces with different sizes are met, and on the premise of not influencing normal operation, the standardization of assembly operation is ensured, the working efficiency is improved, and the equipment investment and management difficulty are reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a bolt assembly detection method, device, computing device and storage medium based on a vision large model. Background Art

[0002] The connection of a ring-shaped bolt group is commonly used in the connection of two pipe or barrel-shaped parts. In actual use, to ensure the tight and reliable connection of the two tubular parts and avoid the leakage of internal gas and liquid, most use a pair of bolts evenly distributed for connection. Therefore, the reliability of the bolt connection has a direct impact on the assembly of the overall product, and the quality of the bolt connection is mainly affected by the bolt assembly process, among which the tightening sequence of the bolts is the key process step. At present, in the mechanical fields at home and abroad, many such connections are manually assembled, and the assembly sequence of the ring-shaped bolt group is also completed by the operator. Due to the large number of bolts in the group, currently the tightening sequence is marked on the bolt holes of the parts, which is time-consuming and laborious and prone to errors. Therefore, it is very necessary to add a monitoring system to guide the workers to install and be able to identify the wrong assembly sequence. Computer vision technology can perform real-time recognition processing and logical judgment on the assembly, which is a necessary means to achieve intelligence.

[0003] The current technology mainly relies on fixing the camera above the mechanical workpiece to obtain an overall view. However, this method has several significant disadvantages. First, the hardware cost is relatively high. Since different sizes and specifications of mechanical workpieces require dedicated fixed cameras, a separate set of placement platforms and shooting equipment must be assembled for each size and specification of the workpiece, which undoubtedly increases the equipment investment of the enterprise. Second, this method lacks flexibility. Especially for those workpieces that are large in volume and difficult to move, adopting this method of fixed cameras results in each workpiece requiring an independent camera, significantly increasing the resource consumption and management cost.

[0004] In addition, the existing methods also have natural inadaptability when dealing with the scenario of a moving camera. Since the shape of the bolt changes with the change of the viewing angle, the existing methods simply rely on the change of pixels before and after to judge whether the worker has tightened the bolt. When directly transplanted to the scenario of a moving camera, the accuracy will be greatly reduced, resulting in an increased risk of misjudgment, thus affecting the overall monitoring effect. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a bolt assembly detection method based on a vision large model to solve the technical defects existing in the prior art. Embodiments of the present invention also provide a bolt assembly detection device based on a vision large model, a computing device, and a computer-readable storage medium.

[0006] According to the first aspect of the embodiments of the present invention, a bolt assembly detection method based on a vision large model is provided, including: Collecting a target image through a wearable device, and identifying visible bolt targets included in the target image through a vision large model; Determining a reference bolt target among the visible bolt targets, and establishing bolt codes corresponding to all the visible bolt targets based on the reference bolt target; Determining a predicted assembly sequence according to the bolt codes, and determining the actual assembly sequence of all the visible bolt targets; Generating bolt assembly information according to the predicted assembly sequence and the actual assembly sequence, and constructing an assembly sequence warning instruction according to the bolt assembly information.

[0007] Optionally, the identifying visible bolt targets included in the target image through a vision large model includes: Identifying the visible bolt targets included in the target image based on the pre-trained vision large model, where the visible bolt targets include a state of a bolt that has been tightened and a state of a bolt that has not been tightened.

[0008] Optionally, the determining a reference bolt target among the visible bolt targets includes: When all the visible bolt targets in the target image are in the state of bolts that have not been tightened, determining the first visible bolt target that changes from the state of a bolt that has not been tightened to the state of a bolt that has been tightened as the reference bolt target.

[0009] Optionally, the establishing bolt codes corresponding to all the visible bolt targets based on the reference bolt target includes: Defining the bolt code corresponding to the reference bolt target as 0; Defining the bolt codes of the remaining visible bolt targets from near to far according to the distance from the reference bolt target, where the value range of the bolt codes is an integer.

[0010] Optionally, after the establishing bolt codes corresponding to all the visible bolt targets based on the reference bolt target, it further includes: Updating the target image based on the image acquisition result of the wearable device; Identifying newly added bolt targets in the target image; Determining the visible bolt targets adjacent to the newly added bolt targets as reference bolt targets; Defining the bolt codes corresponding to the newly added bolt targets based on the bolt codes corresponding to the reference bolt targets.

[0011] Optionally, when the new bolt target is in the state of a bolt that has been tightened, defining the bolt code corresponding to the new bolt target based on the bolt code corresponding to the reference bolt target includes: If the number of reference bolt targets is 1, define the bolt code corresponding to the new bolt target according to the positive and negative values of the bolt code corresponding to the reference bolt target and the positional relationship between the reference bolt target and the new bolt target; If the number of reference bolt targets is 2, calculate the interpolation of the bolt codes corresponding to the 2 reference bolt targets, and define the bolt code corresponding to the new bolt target according to the interpolation calculation result.

[0012] Optionally, when the new bolt target is in the state of a bolt that has not been tightened, defining the bolt code corresponding to the new bolt target based on the bolt code corresponding to the reference bolt target includes: When the reference bolt target has a corresponding non-zero bolt code, define the bolt code corresponding to the new bolt target according to the non-zero bolt code corresponding to the reference bolt target; When the reference bolt target does not have a corresponding non-zero bolt code, define the bolt code corresponding to the new bolt target according to the bolt code corresponding to the reference bolt target that is 0.

[0013] According to the second aspect of the embodiments of the present invention, there is provided a bolt assembly detection device based on a vision large model, including: An identification module, configured to collect a target image through a wearable device and identify visible bolt targets included in the target image through a vision large model; A coding module, configured to determine a reference bolt target among the visible bolt targets and establish bolt codes corresponding to all the visible bolt targets based on the reference bolt target; An order module, configured to determine a predicted assembly order according to the bolt code and determine the actual assembly order of all the visible bolt targets; An alarm module, configured to generate bolt assembly information according to the predicted assembly order and the actual assembly order, and construct an assembly order alarm instruction according to the bolt assembly information.

[0014] According to the third aspect of the embodiments of the present invention, there is provided a computing device, including: A memory and a processor; The memory is used to store computer-executable instructions, and when the processor executes the computer-executable instructions, the steps of the bolt assembly detection method based on the vision large model are implemented.

[0015] According to a fourth aspect of the embodiments of the present invention, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the bolt assembly detection method based on a visual large model.

[0016] According to a fifth aspect of the embodiments of the present invention, there is provided a chip storing a computer program that, when executed by the chip, implements the steps of the bolt assembly detection method based on a visual large model.

[0017] The present invention provides a bolt assembly detection method, device, computing device, and storage medium based on a visual large model. The method collects a target image through a wearable device and identifies visible bolt targets included in the target image through the visual large model; determines a reference bolt target among the visible bolt targets and establishes bolt codes corresponding to all the visible bolt targets based on the reference bolt target; determines a predicted assembly sequence according to the bolt codes and determines the actual assembly sequence of all the visible bolt targets; generates bolt assembly information according to the predicted assembly sequence and the actual assembly sequence, and constructs an assembly sequence warning instruction according to the bolt assembly information. The present invention adapts to the installation requirements of clockwise and counterclockwise orders of workpieces of different sizes, ensures the standardization of assembly operations without affecting normal operations, improves work efficiency, and reduces equipment investment and management difficulty. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for description in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0019] Figure 1 is a flowchart of a bolt assembly detection method based on a visual large model provided by an embodiment of the present invention; Figure 2 is a processing flowchart of a bolt assembly detection method based on a visual large model provided by an embodiment of the present invention; Figure 3 is a bolt schematic diagram of a bolt assembly detection method based on a visual large model provided by an embodiment of the present invention; Figure 4 is a structural schematic diagram of a bolt assembly detection device based on a visual large model provided by an embodiment of the present invention; Figure 5 is a structural block diagram of a computing device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] Numerous specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the present invention. Therefore, the present invention is not limited by the specific implementations disclosed below.

[0021] The terms used in one or more embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of the present invention. The singular forms "a", "the", and "said" used in one or more embodiments of the present invention and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present invention refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0022] In the present invention, a bolt assembly detection method based on a vision large model is provided. The present invention also relates to a bolt assembly detection device based on a vision large model, a computing device, and a computer-readable storage medium, which will be described in detail one by one in the following embodiments.

[0023] Figure 1 The flowchart of a bolt assembly detection method based on a vision large model according to an embodiment of the present invention is shown, which specifically includes the following steps: Step S102: Collect a target image through a wearable device, and identify the visible bolt targets included in the target image through a vision large model; Step S104: Determine the reference bolt target among the visible bolt targets, and establish bolt codes corresponding to all the visible bolt targets based on the reference bolt target; Step S106: Determine the predicted assembly sequence according to the bolt codes, and determine the actual assembly sequence of all the visible bolt targets; Step S108: Generate bolt assembly information according to the predicted assembly sequence and the actual assembly sequence, and construct an assembly sequence alarm instruction according to the bolt assembly information.

[0024] Among them, the visual large model is a model based on deep learning technology, especially a model based on the Transformer architecture, which is used to process and analyze image data. Through the training of a large amount of data, it can automatically extract the feature information in the image, and then realize complex tasks such as image classification, object detection, and image segmentation. The wearable device is an image acquisition device worn by the operator. The image data collected by the image acquisition device is transmitted to the image processing device for processing to realize bolt assembly detection. And the image processing device is integrated in the wearable device, or the image processing device is an independent hardware device that receives the image data sent by the data sending device in the wearable device. The target image is obtained during the operation process after the operator wears the wearable device and contains the image that needs to be subjected to bolt assembly detection.

[0025] Based on this, during the operation process of the operator, the target image is collected through the wearable device, and the image processing device identifies the visible bolt targets therein according to the target image. Then, the first visible bolt target to be assembled is located as the reference bolt target, and all the visible bolt targets are encoded according to the located reference bolt target to obtain bolt encodings corresponding one by one to the visible bolt targets.

[0026] In addition, the predicted assembly sequence is the sequence in which the subsequent visible bolt targets predicted by the image processing device are assembled when the bolts are assembled in a specified clockwise or counterclockwise direction. The actual assembly sequence is the sequence in which the visible bolt targets are actually assembled when the operator performs bolt assembly. Then, after obtaining the bolt encodings, the predicted assembly sequence is determined according to the bolt encodings, and the actual assembly sequence of the operator is monitored through the wearable device. By comparing the predicted assembly sequence with the actual assembly sequence, bolt assembly information is generated. If the bolt assembly information shows that the predicted assembly sequence is the same as the actual assembly sequence, it means that the assembly is correct, and then the assembly sequence alarm instruction will not be sent to the preset alarm device. If the bolt assembly information shows that the predicted assembly sequence is different from the actual assembly sequence, it means that the operator's equipment sequence does not meet the requirements, and then the assembly sequence alarm instruction will be sent to the alarm device for alarm. The alarm methods include lights, sounds, texts, etc. The specific alarm method is determined by the actual use requirements and is not limited in this embodiment.

[0027] Furthermore, in the above step S102, the process of identifying the visible bolt targets included in the target image by the visual large model is specifically implemented as follows in this embodiment: Based on the pre-trained visual large model, the visible bolt targets included in the target image are identified, where the visible bolt targets include the state of bolts that have been tightened and the state of bolts that have not been tightened.

[0028] Among them, the state of the screwed bolt indicates that there is a nut on the stud of the visible bolt target, while the state of the unscrewed bolt indicates that there is no nut on the stud of the visible bolt target. As Figure 2 As shown in the processing flow chart of a bolt assembly detection method based on a vision large model provided, the wearable camera device acquires an image, and then performs instance segmentation to obtain bolts. Among them, the wearable camera device is the wearable device, the image is the target image, and the instance segmentation is implemented by the vision large model using a segmentation algorithm. The obtained bolts are visible bolt targets.

[0029] Furthermore, in the above step S104, the process of determining the reference bolt target among the visible bolt targets is specifically implemented as follows in this embodiment: When all the visible bolt targets in the target image are in the unscrewed bolt state, determine the first visible bolt target that changes from the unscrewed bolt state to the screwed bolt state as the reference bolt target.

[0030] Among them, for the positioning process of the first visible bolt target to be assembled, that is, the reference bolt target, it is necessary to determine that all the visible bolt targets are in the unscrewed bolt state, and the first visible bolt target that changes from the unscrewed bolt state to the screwed bolt state is defined as the reference bolt target.

[0031] Based on this, there is also a situation where the operator leaves midway during the bolt assembly of a certain workpiece. At this time, the first frame of the target image containing visible bolt targets is collected, and the visible bolt targets in it are in both the screwed bolt state and the unscrewed bolt state. Since the operator leaving midway is itself a violation of the operation, at this time, the operator following the memory before leaving for subsequent bolt assembly cannot guarantee the correctness of the assembly sequence. At this time, the operator must disassemble all the bolts and reassemble them. If the operator does not disassemble and chooses to continue the bolt assembly, an assembly sequence alarm instruction is generated to instruct the alarm device to give an alarm.

[0032] Furthermore, in the above step S104, the process of establishing bolt codes corresponding to all visible bolt targets based on the reference bolt target is specifically implemented as follows in this embodiment: Define the bolt code corresponding to the reference bolt target as 0; according to the distance between the reference bolt target and the remaining visible bolt targets, define the bolt codes of the remaining visible bolt targets from near to far, where the value range of the bolt code is an integer.

[0033] Among them, as Figure 2As shown in the processing flow chart of a bolt assembly detection method based on a large vision model, when the first tightened bolt appears, the id is initialized to 0. Subsequently, the distances between other bolts in the field of view and the bolt numbered 0 are calculated, and the bolts on both sides are sorted by distance. The other bolts are manually assigned ids in the order of distance. It should be noted that the first tightened bolt and the bolt numbered 0 are the reference bolt targets, the id is the bolt code, and the other bolts in the field of view are the other visible bolt targets except the reference bolt targets.

[0034] Based on this, in the actual use scenario, in the process of the large vision model using the segmentation algorithm to process the target image, the minimum bounding rectangle of each bolt is used as the initial frame of the multi-target tracking algorithm. The bolt code of the first visible bolt target that becomes the tightened bolt state is defined as 0. The other visible bolt targets are sorted according to the distance from the reference bolt. The farther the distance, the larger the bolt code. For example, the bolt codes of the bolts on the left side of the reference bolt target are assigned negative values, which are -1, -2, -3,... in order from near to far, and all bolt codes in the corresponding counterclockwise order are obtained. Correspondingly, the bolt codes of the bolts on the right side are assigned positive values, which are 1, 2, 3,... in order from near to far, and all bolt codes in the corresponding clockwise order are obtained to adapt to the movement of people in different directions and achieve compatibility with both clockwise and counterclockwise possible operation sequences.

[0035] It should be noted that the relationship that the counterclockwise corresponds to negative bolt codes and the clockwise corresponds to positive bolt codes above can also be that the counterclockwise corresponds to positive bolt codes and the clockwise corresponds to negative bolt codes. The specific corresponding relationship is determined by the actual use scenario and is not limited in this embodiment. In order to ensure that the initial frame of the multi-target tracking algorithm does not lose the visible bolt targets, the hand target of the operator is segmented synchronously, and multiple visible bolt targets are tracked according to the distance between the hand target and the visible bolt targets.

[0036] Furthermore, during the process of the operator assembling bolts, for larger workpieces, there are movement situations. Then, the positions and quantities of the visible bolt targets in the target images collected by the wearable device will change. Some new bolts will appear in the target images, while some old bolts will move out of the target images. At this time, the bolt codes need to be updated. In this embodiment, the specific implementation method of the update process is as follows: Based on the image acquisition result of the wearable device, update the target image; identify the newly added bolt targets in the target image; determine the visible bolt targets adjacent to the newly added bolt targets as the reference bolt targets; define the bolt codes corresponding to the newly added bolt targets based on the bolt codes corresponding to the reference bolt targets.

[0037] Among them, the newly added bolt target is a visible bolt target that does not exist in the target image before the update but exists in the target image after the update. In this case, the newly added bolt target includes two situations. The first situation is that during the bolt assembly process by the operator, the visible bolt target is blocked by the palm, arm, or wrench, etc. The second situation is that during the bolt assembly process by the operator, due to continuous movement, the unassembled bolts enter the target image.

[0038] Based on this, as Figure 2 shown in the processing flow chart of a bolt assembly detection method based on a visual large model provided, a person moves with a camera device to obtain a new image frame, that is, a wearable device for real-time image acquisition, and continuously updates the target image through the acquired image frames. Then, through a multi-object tracking algorithm, if a new bolt appears, obtain the two boundary ids of the currently visible bolts, assign a new bolt id according to the distance, where the new bolt is the newly added bolt target, the two boundary ids of the currently visible bolts are the bolt codes corresponding to the reference bolt targets, and the new bolt id is the bolt code corresponding to the newly added bolt target.

[0039] It should be noted that the method for determining that two reference bolt targets are on both sides of the newly added bolt target is to calculate the center points of the minimum circumscribed rectangles of the three bolts respectively. Let the three points be point A (x1, y1), point B (x2, y2), and point C (x, y), where point C is the center point of the newly added bolt target. Construct vector CB = (x2 - x, y2 - y), vector CA = (x1 - x, y1 - y), and calculate the angle between the vectors. If the angle is greater than 90°, it is considered that the two reference bolt targets are on both sides of the newly added bolt target; otherwise, it is considered to be on the same side.

[0040] Furthermore, if it is the first situation among the above two situations, then in this embodiment, the process of defining the bolt code corresponding to the newly added bolt target based on the bolt code corresponding to the reference bolt target is as follows: If the number of the reference bolt targets is 1, define the bolt code corresponding to the newly added bolt target according to the positive and negative values of the bolt code corresponding to the reference bolt target and the positional relationship between the reference bolt target and the newly added bolt target; if the number of the reference bolt targets is 2, calculate the interpolation of the bolt codes corresponding to the 2 reference bolt targets, and define the bolt code corresponding to the newly added bolt target according to the interpolation calculation result.

[0041] Among them, when the newly added bolt target is a visible bolt target blocked, then this newly added bolt target should be a bolt that has been successfully assembled, so it is in the state of a bolt that has been tightened.

[0042] Based on this, if the number of reference bolt targets is 1, in order to restore the bolt code corresponding to the newly added bolt target, first determine the positive and negative values of the bolt code corresponding to the reference bolt target. If it is positive, then the bolt code corresponding to the newly added bolt target is also positive. On the contrary, if the bolt code of the reference bolt target is negative, then the bolt code corresponding to the newly added bolt target is also negative. Then, according to the positive and negative value judgment result, determine whether the bolt assembly is carried out clockwise or counterclockwise.

[0043] If it is clockwise and the bolt code corresponding to the reference bolt target is positive, then for the newly added bolt target adjacent to the right of the reference bolt target, the corresponding bolt code is the bolt code of the reference bolt target plus 1; if it is clockwise and the bolt code corresponding to the reference bolt target is positive, then for the newly added bolt target adjacent to the left of the reference bolt target, the corresponding bolt code is the bolt code of the reference bolt target plus -1; if it is clockwise and the bolt code corresponding to the reference bolt target is negative, then for the newly added bolt target adjacent to the right of the reference bolt target, the corresponding bolt code is the bolt code of the reference bolt target plus -1; if it is clockwise and the bolt code corresponding to the reference bolt target is negative, then for the newly added bolt target adjacent to the left of the reference bolt target, the corresponding bolt code is the bolt code of the reference bolt target plus 1; if it is counterclockwise and the bolt code corresponding to the reference bolt target is positive, then for the newly added bolt target adjacent to the right of the reference bolt target, the corresponding bolt code is the bolt code of the reference bolt target plus -1; if it is counterclockwise and the bolt code corresponding to the reference bolt target is positive, then for the newly added bolt target adjacent to the left of the reference bolt target, the corresponding bolt code is the bolt code of the reference bolt target plus 1; if it is counterclockwise and the bolt code corresponding to the reference bolt target is negative, then for the newly added bolt target adjacent to the right of the reference bolt target, the corresponding bolt code is the bolt code of the reference bolt target plus 1; if it is counterclockwise and the bolt code corresponding to the reference bolt target is negative, then for the newly added bolt target adjacent to the left of the reference bolt target, the corresponding bolt code is the bolt code of the reference bolt target plus -1. If the number of newly added bolt targets is not 1, then the newly added bolt target with the latest confirmed bolt code can be used as the reference bolt target, and the bolt codes of the newly added bolt targets are assigned in turn.

[0044] In addition, if the number of reference bolt targets is 2, such as Figure 2As shown in the processing flow chart of a bolt assembly detection method based on a large vision model, if there is an occluded target bolt with its ID lost and then reappearing, the ID of the target bolt is interpolated using the IDs of the two adjacent bolts on both sides to reassign an ID to it. Specifically, the bolt code of the newly added bolt target is the integer between the bolt codes of these two reference bolt targets. Through the interpolation method, the bolt code of the newly added bolt target can be determined. In addition, it should be noted that when the number of newly added bolt targets is not 1, it is necessary to ensure that the numerical arrangement of the bolt codes of the newly added bolt targets determined by the interpolation method is the same as that of the bolt codes of the reference bolt targets.

[0045] Furthermore, for the second case, the process of defining the bolt code corresponding to the newly added bolt target based on the bolt code corresponding to the reference bolt target is as follows in this embodiment: When the reference bolt target has a non-zero bolt code corresponding to it, define the bolt code corresponding to the newly added bolt target according to the non-zero bolt code corresponding to the reference bolt target; When the reference bolt target does not have a non-zero bolt code corresponding to it, define the bolt code corresponding to the newly added bolt target according to the bolt code equal to 0 corresponding to the reference bolt target.

[0046] Among them, since the newly added bolt target is an unassembled bolt, the newly added bolt target is in the state of an unfastened bolt. At this time, there are also two cases. If there is a non-zero bolt code among the bolt codes corresponding to the reference bolt targets, in this case, it is necessary to define the bolt code of the newly added bolt target according to this non-zero bolt code of the reference bolt target. The specific definition method refers to the method when the newly added bolt target is in the state of a fastened bolt and the number of reference bolt targets is 1, which will not be elaborated in this embodiment. This avoids the situation where after the operator assembles the bolts for a week, at the end stage, there are two reference bolt targets on the target image, one of which is the reference bolt target, and defining the bolt code of the newly added bolt target according to the bolt code of the reference bolt target results in a jump in the values between all the bolt codes.

[0047] For example, as Figure 3 As shown in the bolt schematic diagram of a bolt assembly detection method based on a large vision model provided, the visible bolt targets included in the target image of the previous frame have corresponding bolt codes of 2 - 5. The newly emerged bolt is the newly added bolt target in the target image of the current frame. At this time, the reference bolt target is determined to be the visible bolt target with a bolt code of 5. Then, when it is judged that its bolt code is a positive value, the bolt code 5 is incremented by 1 to obtain 6, which is the bolt code corresponding to the newly added bolt target.

[0048] In addition, if there is only a bolt code of 0 for the reference bolt target, then it is necessary to define the bolt code for the newly added bolt target according to the bolt code of this reference bolt target with a value of 0. For the specific definition method, refer to the method described above where the newly added bolt target is in the tightened bolt state and the number of reference bolt targets is 1. This will not be elaborated in this embodiment.

[0049] After that, the execution processes of steps S106 and S108 are as Figure 2 shown in the processing flow chart of a bolt assembly detection method based on a vision large model. If there is a new tightened bolt, record its id and compare it with the result of predicting the id of the next tightened bolt based on the list of tightened bolt ids. If the new tightened bolt is consistent with the predicted id, that is, "the id is in the predicted ids", then it proves that the bolt assembly sequence is correct, and update the id of the new tightened bolt to the list of tightened bolt ids. If they are inconsistent, then it proves that the bolt assembly sequence is incorrect and an alarm is required.

[0050] Moreover, the judgment of the completion of the assembly of all bolts on a workpiece is as Figure 2 shown. When the judgment condition for the completion of one circle of bolts being tightened is met, if it is met, end the current task; if not, continue the detection. Among them, the judgment condition for the completion of one circle of bolts being tightened is that when there is only one visible bolt target in the untightened bolt state in the target image, and the two visible bolt targets closest to it are both in the tightened bolt state, and these two visible bolt targets in the tightened bolt state are located on both sides of the visible bolt target in the untightened bolt state, it is determined that the visible bolt target in the current untightened bolt state is the last untightened bolt. When it becomes in the tightened bolt state, it is considered that one circle of bolts in the annular bolt group has been tightened, and the current task is ended.

[0051] Corresponding to the above method embodiment, the present invention also provides an embodiment of a bolt assembly detection device based on a vision large model. Figure 4 The structural schematic diagram of a bolt assembly detection device based on a vision large model provided by an embodiment of the present invention is shown. As Figure 4 shown, the device includes: An identification module 402, configured to collect a target image through a wearable device and identify the visible bolt targets included in the target image through a vision large model; An encoding module 404, configured to determine a reference bolt target among the visible bolt targets and establish bolt codes corresponding to all the visible bolt targets based on the reference bolt target; A sequence module 406, configured to determine a predicted assembly sequence according to the bolt codes and determine the actual assembly sequence of all the visible bolt targets; An alarm module 408, configured to generate bolt assembly information based on the predicted assembly sequence and the actual assembly sequence, and construct an assembly sequence alarm instruction based on the bolt assembly information.

[0052] In an alternative embodiment, the recognition module 402 is further configured to: Based on the pre-trained visual large model, recognize the visible bolt targets included in the target image, where the visible bolt targets include the state of tightened bolts and the state of untightened bolts.

[0053] In an alternative embodiment, the encoding module 404 is further configured to: When all the visible bolt targets in the target image are in the state of untightened bolts, determine the first visible bolt target that changes from the state of untightened bolts to the state of tightened bolts as the reference bolt target.

[0054] In an alternative embodiment, the encoding module 404 is further configured to: Define the bolt code corresponding to the reference bolt target as 0; according to the distance between the reference bolt target and the remaining visible bolt targets, define the bolt codes of the remaining visible bolt targets from near to far, where the value range of the bolt code is an integer.

[0055] In an alternative embodiment, the bolt assembly detection device based on the visual large model further includes: An update module, configured to update the target image based on the image acquisition result of the wearable device; recognize the newly added bolt targets in the target image; determine the visible bolt targets that are adjacent to the newly added bolt targets and are in the state of tightened bolts as the reference bolt targets; define the bolt code corresponding to the newly added bolt target based on the bolt code corresponding to the reference bolt target.

[0056] In an alternative embodiment, the update module is further configured to: When the newly added bolt target is in the state of tightened bolts, if the number of the reference bolt targets is 1, define the bolt code corresponding to the newly added bolt target according to the positive or negative value of the bolt code corresponding to the reference bolt target and the positional relationship between the reference bolt target and the newly added bolt target; if the number of the reference bolt targets is 2, calculate the interpolation of the bolt codes corresponding to the 2 reference bolt targets, and define the bolt code corresponding to the newly added bolt target according to the interpolation calculation result.

[0057] In an alternative embodiment, the update module is further configured to: When the new bolt target is in the state of an untightened bolt, when the reference bolt target has a corresponding non-zero bolt code, according to the non-zero bolt code corresponding to the reference bolt target, define the bolt code corresponding to the new bolt target; when the reference bolt target does not have a corresponding non-zero bolt code, according to the bolt code corresponding to the reference bolt target that is 0, define the bolt code corresponding to the new bolt target.

[0058] The bolt assembly detection device based on a visual large model provided by the present invention collects a target image through a wearable device, and identifies visible bolt targets included in the target image through the visual large model; determines a reference bolt target among the visible bolt targets, and establishes bolt codes corresponding to all the visible bolt targets based on the reference bolt target; determines a predicted assembly sequence according to the bolt codes, and determines the actual assembly sequence of all the visible bolt targets; generates bolt assembly information according to the predicted assembly sequence and the actual assembly sequence, and constructs an assembly sequence warning instruction according to the bolt assembly information. The present invention meets the installation requirements of clockwise and counterclockwise sequences for workpieces of different sizes, ensures the standardization of assembly operations, improves work efficiency, and reduces equipment investment and management difficulty without affecting normal operations.

[0059] The above is a schematic solution of a bolt assembly detection device based on a visual large model according to an embodiment of the present invention. It should be noted that the technical solution of the bolt assembly detection device based on a visual large model and the technical solution of the bolt assembly detection method based on a visual large model described above belong to the same concept. For the details not described in detail in the technical solution of the bolt assembly detection device based on a visual large model, reference can be made to the description of the technical solution of the bolt assembly detection method based on a visual large model described above. In addition, each component in the device embodiment should be understood as a functional module that must be established to implement each step of the program flow or each step of the method. Each functional module is not an actual functional division or separation limitation. The device claim defined by such a set of functional modules should be understood as mainly implementing the functional module framework of the solution through the computer program recorded in the specification, rather than mainly implementing the physical device of the solution through hardware means.

[0060] Figure 5 The structural block diagram of a computing device 500 according to an embodiment of the present invention is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 through a bus 530, and a database 550 is used to store data.

[0061] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of such networks include the Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0062] In one embodiment of the present invention, the above components of the computing device 500 and Figure 5 other components not shown may also be connected to each other, for example, via a bus. It should be understood that Figure 5 the block diagram of the computing device shown is for illustrative purposes only and is not a limitation on the scope of the present invention. Those skilled in the art can add or replace other components as needed.

[0063] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 500 can also be a mobile or stationary server.

[0064] Wherein, the processor 520 is used to execute the computer-executable instructions for each step of the bolt assembly detection method based on the visual large model.

[0065] The above is a schematic solution of a computing device in this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above bolt assembly detection method based on the visual large model belong to the same concept. For the details not described in the technical solution of the computing device, reference can be made to the description of the technical solution of the above bolt assembly detection method based on the visual large model.

[0066] An embodiment of the present invention also provides a computer-readable storage medium, which stores computer instructions that, when executed by a processor, are used to execute each step of the bolt assembly detection method based on the visual large model.

[0067] The above is a schematic solution of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above bolt assembly detection method based on a visual large model belong to the same concept. For the details not described in the technical solution of the storage medium, reference can be made to the description of the technical solution of the above bolt assembly detection method based on a visual large model.

[0068] An embodiment of the present invention further provides a chip, which stores a computer program. When the computer program is executed by the chip, the steps of the bolt assembly detection method based on the visual large model are implemented.

[0069] The above describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0070] The computer instructions include computer program code, which may be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0071] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily all essential to the present invention.

[0072] In the above embodiments, the descriptions of each embodiment have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0073] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The alternative embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, according to the content of the present invention, many modifications and variations can be made. The present invention selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can well understand and utilize the present invention. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. A bolt assembly detection method based on a large vision model, characterized in that, Including: Collect a target image through a wearable device, and identify visible bolt targets included in the target image through a visual large model; Determine a reference bolt target among the visible bolt targets, and establish bolt codes corresponding to all the visible bolt targets based on the reference bolt target; Determine a predicted assembly sequence according to the bolt codes, and determine the actual assembly sequence of all the visible bolt targets; Generate bolt assembly information according to the predicted assembly sequence and the actual assembly sequence, and construct an assembly sequence warning instruction according to the bolt assembly information.

2. The method according to claim 1, characterized in that, The identifying visible bolt targets included in the target image through the visual large model includes: Based on the pre-trained visual large model, identify the visible bolt targets included in the target image, where the visible bolt targets include the state of tightened bolts and the state of untightened bolts.

3. The method according to claim 2, wherein The determining a reference bolt target among the visible bolt targets includes: When all the visible bolt targets in the target image are in the state of untightened bolts, determine the first visible bolt target that changes from the state of untightened bolts to the state of tightened bolts as the reference bolt target.

4. The method according to claim 1, wherein The establishing bolt codes corresponding to all the visible bolt targets based on the reference bolt target includes: Define the bolt code corresponding to the reference bolt target as 0; Define the bolt codes of the remaining visible bolt targets from near to far according to the distance between them and the reference bolt target, where the value range of the bolt codes is an integer.

5. The method according to claim 2, wherein After establishing bolt codes corresponding to all the visible bolt targets based on the reference bolt target, it further includes: Update the target image based on the image acquisition result of the wearable device; Identify newly added bolt targets in the target image; Determine the visible bolt target adjacent to the newly added bolt target as a reference bolt target; Define the bolt code corresponding to the newly added bolt target based on the bolt code corresponding to the reference bolt target.

6. The method according to claim 5, wherein When the newly added bolt target is in the state of tightened bolts, the defining the bolt code corresponding to the newly added bolt target based on the bolt code corresponding to the reference bolt target includes: If the number of reference bolt targets is 1, define the bolt code corresponding to the newly added bolt target according to the positive or negative value of the bolt code corresponding to the reference bolt target and the positional relationship between the reference bolt target and the newly added bolt target; If the number of reference bolt targets is 2, calculate the interpolation of the bolt codes corresponding to the 2 reference bolt targets, and define the bolt code corresponding to the newly added bolt target according to the interpolation calculation result.

7. The method according to claim 5, wherein When the newly added bolt target is in the state of untightened bolts, the defining the bolt code corresponding to the newly added bolt target based on the bolt code corresponding to the reference bolt target includes: When there is a non-zero bolt code corresponding to the reference bolt target, define the bolt code corresponding to the newly added bolt target according to the non-zero bolt code corresponding to the reference bolt target. When there is no corresponding non-zero bolt code for the reference bolt target, define the bolt code corresponding to the newly added bolt target according to the bolt code that is 0 corresponding to the reference bolt target.

8. A bolt assembly detection device based on a large vision model, characterized in that, Comprising: An identification module, configured to collect a target image through a wearable device and identify visible bolt targets included in the target image through a vision large model; A coding module, configured to determine a reference bolt target among the visible bolt targets and establish bolt codes corresponding to all the visible bolt targets based on the reference bolt target; A sequence module, configured to determine a predicted assembly sequence according to the bolt codes and determine the actual assembly sequence of all the visible bolt targets; An alarm module, configured to generate bolt assembly information according to the predicted assembly sequence and the actual assembly sequence, and construct an assembly sequence alarm instruction according to the bolt assembly information.

9. A computing device, characterized in that, Comprising: A memory and a processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the bolt assembly detection method based on a vision large model according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer instructions, characterized in that, When the instruction is executed by the processor, the steps of the bolt assembly detection method based on a vision large model according to any one of claims 1 to 7 are implemented.

Images