Automobile work island assembly method, system, storage medium and electronic equipment

Abstract

The present invention discloses an automobile work island assembly method, system, storage medium and electronic equipment. The method comprises: installing at least one image acquisition device on each workstation of the automobile work island, the image acquisition device capturing image data of workers and their working environment in real time; using an image processing algorithm to process the image data, identifying the actions and postures of the workers, extracting key features, and generating feature data of the workers' assembly; constructing a state assessment model based on machine learning based on pre-labeled historical feature data; inputting the feature data obtained by the current image processing into the state assessment model for real-time assessment, generating assessment results of the worker state and assembly state, sending the assessment results and related image data to the work island assembly control terminal, controlling the assembly quality of the automobile work island, providing real-time and accurate worker state monitoring for automobile manufacturing companies, and improving assembly quality and work efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of automobile assembly, and in particular to an automobile work island assembly method, system, storage medium and electronic equipment. Background Art

[0002] With the rapid development of the global automotive industry, the improvement of production efficiency and product quality has become an important factor in corporate competition. In the automobile manufacturing process, the assembly link is a key step to ensure the quality of the final product. The traditional automobile work island assembly method relies on manual operation and supervision by workers. Although it can meet production needs to a certain extent, it faces many challenges as the production scale expands and the product complexity increases.

[0003] First, the high intensity, high frequency and repetitive work of manual operation can easily lead to operator fatigue, thus increasing the risk of incorrect operation. This not only affects the efficiency of assembly, but may also have a negative impact on assembly quality. For example, workers may ignore certain key steps due to fatigue during the assembly process, or incorrectly install assembly parts due to incorrect posture, which may cause customer complaints and rework, thus affecting brand reputation and cost control. Summary of the invention

[0004] The purpose of the present invention is to provide an automobile work island assembly method, system, storage medium and electronic equipment to solve the deficiencies in the prior art. It is an intelligent assembly method that can integrate image acquisition and advanced data processing technologies, and can provide automobile manufacturers with real-time and accurate worker status monitoring, thereby improving assembly quality and work efficiency. It has important practical significance and market demand.

[0005] An embodiment of the present application provides an automobile work island assembly method, the method comprising:

[0006] At least one image acquisition device is installed at each workstation of the automobile work island, and the image acquisition device can capture image data of workers and their working environment in real time, and the image data includes the worker's operating actions, working postures, tool usage and the status of assembled parts;

[0007] Processing the image data using an image processing algorithm to identify the worker's movements and postures and extract key features to generate feature data of the worker's assembly, wherein the key features include the worker's hand position and the status of the tool;

[0008] Based on the pre-labeled historical feature data, a machine learning-based state assessment model is constructed, which is used to assess the worker's current operating state and assembly state, where the operating state includes effective operation, potential risk operation, and invalid operation, and the assembly state includes not started, in progress, and completed;

[0009] The feature data obtained by the current image processing is input into the state assessment model for real-time assessment, and the assessment results of the worker status and assembly status are generated. The assessment results and related image data are sent to the work island assembly control terminal to manage the assembly quality of the automobile work island, wherein the assessment results include status labels and operation suggestions.

[0010] Optionally, the image data is processed using an image processing algorithm to identify the worker's movements and postures and extract key features to generate feature data of the worker's assembly, including:

[0011] Using an accelerated robust feature algorithm to detect key feature points in the image, applying an ORB algorithm to describe the key feature points, and generating feature descriptors, wherein the key feature points include the worker's body feature points and tool feature points;

[0012] The extracted key feature points and their feature descriptors are input into the pre-trained motion classification model to identify the worker's hand movements. At the same time, the posture information of the worker's specific joints is obtained by combining the posture estimation algorithm. The posture information and feature descriptors are used together to analyze whether the worker's posture meets the optimal process standard.

[0013] According to the tool status, feature descriptors are used to realize tool feature comparison, wherein the tool outline is identified by feature matching algorithm, and the generated feature descriptors are used for matching to determine whether the tool is in the correct use state, incorrect use state or standby state;

[0014] The extracted hand movements, posture information, and tool status are integrated into a structured feature set to obtain the feature data of the worker's assembly.

[0015] Optionally, the state assessment model based on machine learning is constructed based on the pre-labeled historical feature data, and the model is used to assess the current operation state and assembly state of the worker, including:

[0016] Collect historical image datasets from the assembly process of automobile work islands, which cover various operations and assembly situations. The data sources include operations of different workstations and different workers, as well as the use of various tools and parts.

[0017] The historical feature dataset of workers' assembly is extracted from the historical image set, and the state assessment model is trained using support vector machine, random forest or deep learning model to obtain a trained state assessment model. In this model, a multi-task learning framework is used to simultaneously train the workers' operation state and assembly state, and mutual learning between different tasks is promoted by sharing the underlying feature representation.

[0018] Optionally, the feature data obtained by the current image processing is input into the state assessment model for real-time assessment, and assessment results of the worker state and the assembly state are generated, and the assessment results and related image data are sent to the work island assembly control terminal to control the assembly quality of the automobile work island, including:

[0019] The feature data obtained by the current image processing is input into the state assessment model for real-time assessment, and a series of forward propagation calculations are performed to generate real-time assessment results, wherein the assessment results include state labels and operation suggestions. The state labels will clarify the current operation state and assembly state of the worker, and the operation suggestions provide targeted improvement measures, which include adjusting the worker's operation method or changing the use of tools.

[0020] Optionally, the method further includes:

[0021] The environmental data of the automobile work island and the behavioral data of workers related to the environment during the assembly process are collected to calculate the environmental suitability index used to reflect the impact of the environment on assembly.

[0022] Another embodiment of the present application provides an automobile work island assembly system, the system comprising:

[0023] A capture module is used to install at least one image acquisition device at each workstation of the automobile work island, and the image acquisition device can capture image data of workers and their working environment in real time, and the image data includes the worker's operating actions, working postures, tool usage and the status of assembled parts;

[0024] An extraction module, used to process the image data using an image processing algorithm, identify the worker's movements and postures, and extract key features to generate feature data of the worker's assembly, wherein the key features include the worker's hand position and the status of the tool;

[0025] A construction module is used to construct a state assessment model based on machine learning based on pre-labeled historical feature data, wherein the model is used to assess the current operation state and assembly state of the worker, wherein the operation state includes effective operation, potential risk operation and invalid operation, and the assembly state includes not started, in progress and completed;

[0026] The evaluation module is used to input the feature data obtained by the current image processing into the state evaluation model for real-time evaluation, generate evaluation results of the worker status and assembly status, and send the evaluation results and related image data to the work island assembly control terminal to manage the assembly quality of the automobile work island, wherein the evaluation results include status labels and operation suggestions.

[0027] Yet another embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, wherein the computer program is configured to execute any of the above methods when running.

[0028] Yet another embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any of the methods described above.

[0029] Compared with the prior art, the present invention provides an automobile work island assembly method, in which at least one image acquisition device is installed on each workstation of the automobile work island, and the image acquisition device can capture image data of workers and their working environment in real time; an image processing algorithm is used to process the image data, identify the workers' movements and postures, and extract key features to generate feature data of the workers' assembly; based on pre-labeled historical feature data, a state assessment model based on machine learning is constructed; the feature data obtained by the current image processing is input into the state assessment model for real-time assessment, and assessment results of the worker status and assembly status are generated, and the assessment results and related image data are sent to the work island assembly control terminal to manage the assembly quality of the automobile work island, thereby being able to integrate image acquisition and advanced data processing technologies into an intelligent assembly method, which can provide automobile manufacturers with real-time and accurate worker status monitoring, thereby improving assembly quality and work efficiency, and has important practical significance and market demand. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A hardware structure block diagram of a computer terminal for an automobile work island assembly method provided by an embodiment of the present invention;

[0031] Figure 2 A schematic flow chart of an automobile work island assembly method provided by an embodiment of the present invention;

[0032] Figure 3 A schematic structural diagram of an automobile work island assembly system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, but should not be construed as limiting the present invention.

[0034] The embodiment of the present invention firstly provides a method for assembling an automobile work island, which can be applied to electronic equipment, such as a computer terminal, specifically a common computer, etc.

[0035] The following describes it in detail by taking running on a computer terminal as an example. Figure 1The hardware structure block diagram of a computer terminal of a vehicle work island assembly method provided by an embodiment of the present invention. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the computer terminal may also include a transmission device 106 for communication functions and an input and output device 108. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0036] The memory 104 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the automobile work island assembly method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0038] See also Figure 2 , an embodiment of the present invention provides an automobile work island assembly method, which may include the following steps:

[0039] S201, installing at least one image acquisition device at each workstation of the automobile work island, wherein the image acquisition device can capture image data of workers and their working environment in real time, wherein the image data includes the workers' operating actions, working postures, tool usage, and status of assembled parts;

[0040] By installing image acquisition equipment, comprehensive monitoring and data collection of workers and their working environment can be achieved. Image acquisition equipment can record every operation detail of workers during the assembly process, including their movements, postures, types and status of tools used, and the quality and progress of the parts being assembled. The resulting image data not only provides a basis for subsequent data analysis, but also provides a real-time and objective source of information for building an intelligent monitoring system to ensure the standardization and scientific nature of the assembly process.

[0041] Specifically, you can choose high-resolution cameras (such as RGB-D cameras or industrial cameras) and 360-degree panoramic cameras to ensure that every corner of the workstation is covered, and have night vision functions to adapt to different lighting conditions. Arrange multiple cameras at each workstation to ensure that the worker's operation is captured from different angles to avoid blind spots. At the same time, configure motion sensors to automatically adjust the camera's focus and viewing angle to follow the worker's movements.

[0042] Use edge computing devices to process video streams in real time and quickly transfer data to local servers to reduce latency. Edge devices can upload data to the cloud or local storage in real time through local networks for subsequent analysis. Ensure stable image capture under high load conditions and avoid data loss by setting up multi-channel parallel processing. Use machine learning models (such as YOLO or Mask R-CNN) to automatically identify and annotate workers' movements, tool use, and part status. Automatically generate labels based on previously trained models to reduce the burden of manual annotation. Ensure that image acquisition equipment can capture background information of the workstation, such as tool sheds and parts storage locations, to help with subsequent analysis of the interaction between workers and the environment.

[0043] Use optical flow or posture estimation algorithms (such as OpenPose) to analyze the worker's motion trajectory and posture, extract the hand position and tool usage, and generate feature data sets. Through image comparison and machine learning algorithms, the status of the tool can be evaluated in real time to determine whether it is used correctly and capture the status of the assembly parts.

[0044] An efficient data management system can be set up to classify and store the real-time captured image data with timestamps and workstation numbers for easy subsequent retrieval and analysis. Encryption technology is used to protect data to ensure the security and privacy of image data and prevent unauthorized access and data leakage.

[0045] S202, using an image processing algorithm to process the image data, identify the worker's movements and postures, and extract key features to generate feature data of the worker's assembly, wherein the key features include the worker's hand position and the status of the tool;

[0046] Image processing algorithms are used to extract key information from captured image data. Advanced computer vision technology is used to identify workers’ movements and postures, and extract feature data related to assembly operations. For example, the position and posture of workers’ hands during the assembly process, the status of the tools used, etc. These feature data will be used in subsequent status assessment models to help determine the effectiveness and potential risks of workers in the assembly process.

[0047] Specifically, the accelerated robust feature algorithm can be used to detect key feature points in the image, and the ORB algorithm can be applied to describe the key feature points to generate feature descriptors, wherein the key feature points include the worker's body feature points and tool feature points;

[0048] First, the SURF algorithm is used to detect key feature points from the image. These feature points can effectively represent the structural information in the image. Then, these feature points are described by the ORB (Oriented FAST and Rotated BRIEF) algorithm to generate feature descriptors. The feature descriptors will provide specific feature information of these key points, making the subsequent matching and recognition processes more efficient and accurate.

[0049] The accelerated robust feature algorithm can quickly extract key feature points while maintaining high accuracy, improving the real-time performance of image processing and contributing to the effective operation of the real-time monitoring system. By generating feature descriptors, feature matching can be effectively performed in subsequent tool identification and worker motion analysis, improving the recognition rate and accuracy of the system. The combination of SURF and ORB algorithms can effectively resist the effects of lighting changes and perspective changes, making the system stable in a variety of working environments.

[0050] First, high-performance cameras are used to collect image data of the workers' working environment. These images need to have high resolution to capture fine details. These image data are input into the SURF algorithm, which can effectively identify key feature points by generating a multi-scale image pyramid for analysis. SURF can handle image transformations (such as rotation, scaling) and lighting changes to ensure the robustness of key points. Through the calculation of the Hessian matrix of SURF, feature points with high response values ​​are detected. These points are considered to be significant features in the image, namely key feature points.

[0051] For the detected key feature points, the ORB algorithm is used to generate feature descriptors. ORB first uses the FAST corner detection algorithm to quickly extract the key points in the image, then calculates the direction of these points, and finally generates unchanged feature descriptors. The feature descriptor is a vector, which is calculated by the BRIEF (Binary Robust Invariant ScalableKeypoints) feature descriptor, which can provide an efficient and compact binary encoding for each feature point, so that the subsequent matching process can be carried out quickly. The descriptors of the key feature points will be stored in a feature database for subsequent matching and recognition.

[0052] The extracted key feature points and their feature descriptors are input into the pre-trained motion classification model to identify the worker's hand movements. At the same time, the posture information of the worker's specific joints is obtained by combining the posture estimation algorithm. The posture information and feature descriptors are used together to analyze whether the worker's posture meets the optimal process standard.

[0053] The extracted key feature points and their descriptors are passed to a trained motion classification model to identify the worker's hand movements. At the same time, combined with the posture estimation algorithm, the posture information of certain specific joints of the worker's body can be obtained. The combination of these two aspects of information helps to analyze whether the worker's overall posture meets the best process standards. Through the comprehensive analysis of hand movements and body joint postures, the worker's working status can be more comprehensively evaluated, thereby improving the intelligence level of the monitoring system; analyzing whether the worker's posture meets the best process standards can promote the implementation of training and improvement measures, thereby improving assembly quality and safety; combining motion recognition with posture information provides data support for subsequent adaptive control systems, and can dynamically adjust the workflow based on the worker's real-time operation feedback.

[0054] You can choose a deep learning-based motion classification model, such as a convolutional neural network (CNN), which is trained with a large amount of labeled data of worker motions and can classify hand motions. The model structure needs to be optimized to improve its efficiency in real-time processing. The extracted feature descriptors are used as input and input into the motion classification model after preprocessing (such as normalization). Based on the motion classification, a pose estimation algorithm (such as OpenPose) is used to obtain the position information of specific joints of the worker's body and build a complete body joint model. The pose estimation algorithm can generate a pose diagram of the worker's body by analyzing the coordinate system of each key point in the image and extract the angle and position of each joint. The motion recognition results are combined with the pose information to generate a comprehensive report by comparing the actual movements of the worker with the best process standards. This report will provide a specific assessment to identify whether the worker has non-standard postures or movements in the operation.

[0055] According to the tool status, feature descriptors are used to realize tool feature comparison, wherein the tool outline is identified by feature matching algorithm, and the generated feature descriptors are used for matching to determine whether the tool is in the correct use state, incorrect use state or standby state;

[0056] Use feature descriptors to compare the status of the tool. Through feature matching algorithms, the system can identify the outline of the tool and match it with known tool features. This can determine the current usage status of the tool, including whether it is being used correctly, incorrectly, or in standby. Real-time monitoring of the tool usage status can quickly identify improper operations, thereby reducing quality problems caused by improper tool use; by identifying the status of the tool, it can optimize tool management and allocation, and improve tool utilization efficiency during the assembly process; timely identification of incorrect tool use and providing workers with rectification suggestions will help ensure operational safety and reduce the possibility of accidents.

[0057] The tool image can be processed by using the Canny edge detection algorithm. The tool contour can be extracted by setting high and low thresholds to obtain the tool boundary information. The generated contour data will be used for subsequent feature matching. The extracted tool contour is compared with the tool feature descriptor stored in the feature database, and the FLANN (Fast Library for Approximate Nearest Neighbors) algorithm is used for feature matching. FLANN can quickly and effectively find the nearest feature point to ensure the efficiency of the matching process. In the feature matching process, the tool status is judged by combining distance metrics (such as Euclidean distance). Set a threshold to distinguish between "correct use status", "incorrect use status" and "standby status". According to the matching results, the tool status information is generated. If the matching degree is higher than the threshold, it is judged as "correct use status"; if it is lower than the threshold but higher than another set value, it is judged as "standby status"; if it is lower than the minimum threshold, it is judged as "incorrect use status". The tool status can also be fed back to the control terminal to provide workers with immediate operation suggestions.

[0058] The extracted hand movements, posture information, and tool status are integrated into a structured feature set to obtain the feature data of the worker's assembly.

[0059] This step aims to integrate the hand movements, worker postures, and tool states extracted from the image into a structured feature set. This data format facilitates subsequent analysis and processing, and also allows different types of data to be processed and interpreted uniformly. Through structured feature sets, multiple data types can be processed uniformly, thereby improving the efficiency and accuracy of data analysis; structured data sets provide standardized inputs for subsequent machine learning models, making model training and prediction more efficient; the integrated feature data provides management with rich information, enabling more scientific and reasonable judgments based on data when making decisions.

[0060] The extracted hand movements, posture information, and tool status can be standardized and a unified data structure, such as JSON format, can be defined to ensure the versatility and readability of the data in subsequent processing. - Determine the key fields in the feature set, such as "hand movements", "joint postures", "tool status", etc., and define each field in detail. Fill all the extracted information into a standardized data structure to form a complete feature data set. The data structure should contain a timestamp to record the time information of each state for subsequent timing analysis. Through the feature extraction system, this structured feature set is stored in the database for real-time access and subsequent data analysis.

[0061] After the feature set is integrated, data verification can also be performed to ensure the accuracy and consistency of the data through cross-checking. Data integrity checks are used to ensure that all key information has been extracted and integrated. Data cleaning and updating are performed regularly to maintain the timeliness and effectiveness of the feature data set, providing a reliable data foundation for subsequent statistical analysis and report generation.

[0062] Through the above steps, an efficient, accurate and real-time updateable worker assembly status monitoring system can be formed, which can effectively improve the quality control and management level of the automobile work island assembly process.

[0063] S203, based on the pre-labeled historical feature data, construct a state assessment model based on machine learning, wherein the model is used to assess the worker's current operation state and assembly state, wherein the operation state includes effective operation, potential risk operation and invalid operation, and the assembly state includes not started, in progress and completed;

[0064] 1. Pre-annotated historical feature data: This dataset is composed of past worker operation and assembly process records, annotated by professionals, and contains samples of various worker behaviors, as well as operation and assembly states in different situations. These annotations can be descriptions of workers’ actions, tools used, work efficiency, and the state of assembled parts, etc.

[0065] 2. Build a state assessment model based on machine learning: By using machine learning algorithms (such as support vector machines, random forests, etc.) to train historical feature data, the model can learn the relationship between different workers' operating states and assembly states. This means that the model can automatically assess the current state of the worker based on the input feature data.

[0066] 3. Evaluate the current operating status and assembly status of workers: The model not only evaluates whether workers are performing their tasks effectively, but also identifies potential operating risks. This is crucial to the safety and efficiency of the production process. For example, if a worker's operation is evaluated as a "potential risk operation", a warning can be issued in time and measures can be taken to avoid accidents.

[0067] 4. Classification of operation status and assembly status: The classification of operation status helps monitor the behavior of workers. Effective operation means that workers are working according to best practices; potential risk operation means that workers may have improper operation methods; invalid operation shows that workers fail to work as expected. The classification of assembly status reflects the progress of the assembly process. By understanding the current assembly status (not started, in progress, completed), managers can better allocate resources and arrange work progress.

[0068] Specifically, a historical image dataset from the assembly process of an automobile work island may be collected, wherein the historical image dataset covers various operations and assembly conditions, and the data sources include operations of different workstations and different workers, and the use of various tools and parts;

[0069] This process involves collecting image data from production lines and workstations during various operations and assembly processes. This data includes not only the actions of workers, but also the use of different workstations, tools, and assembly parts. With diverse data sources, the model can learn more comprehensive features. Ensure that the acquired data is representative and can cover various situations that may occur in a real work environment, thereby improving the generalization ability of the model.

[0070] First, the team can be organized to conduct on-site filming at different automobile work island assembly lines to ensure that the operation process of multiple workstations is covered. Use high-resolution camera equipment to shoot the entire process of workers in assembly, and the duration and angle should be diversified to capture different operation scenarios. The collected image data needs to be annotated by professionals, and the annotation content includes the worker's behavior (such as using tools, installing parts, etc.), workstation settings, tool types, assembly status, etc., to ensure that the data has high-quality label information. This process can be carried out using image annotation tools (such as LabelMe, VGG Image Annotator, etc.). The processed images and their related label information are classified and stored in the database according to attributes such as time, workstation, and worker, and structured file naming rules are used to facilitate subsequent retrieval and use. This can be achieved using SQL databases or NoSQL databases (such as MongoDB) to ensure efficient storage and access of data.

[0071] The historical feature dataset of workers' assembly is extracted from the historical image set, and the state assessment model is trained using support vector machine, random forest or deep learning model to obtain a trained state assessment model. In this model, a multi-task learning framework is used to simultaneously train the workers' operation state and assembly state, and mutual learning between different tasks is promoted by sharing the underlying feature representation.

[0072] Extract key features from the collected historical image data. These features can be the actions of workers during operation, the use of tools, and the status of assembled parts. Select support vector machines, random forests, deep learning and other algorithms to train the model. These methods are highly applicable and can extract effective information from complex data. Through the multi-task learning framework, the model can learn the worker's operating status and assembly status at the same time, so that the learning of the two complements each other. Such training can improve the accuracy and robustness of the model, can effectively identify and classify the worker's current operating and assembly status, and improve the efficiency and accuracy of real-time evaluation.

[0073] Convolutional neural network (CNN) can be used as the basic model for feature extraction. The collected image data is input into CNN, and the network extracts high-level features through multiple layers of convolution and pooling operations. Select a pre-trained network (such as ResNet or VGG) and fine-tune it to adapt to the specific assembly scenario. Based on feature extraction, a multi-task learning framework is constructed. The framework consists of a shared underlying feature extraction network and two branch networks. One branch network is used to predict the worker's operating status, and the other branch is used to predict the assembly status. The outputs of both are trained separately using the cross entropy loss function, while the model's learning ability for different tasks is enhanced by sharing the underlying features. The model is trained using annotated image datasets, and the loss function is optimized using the Adam optimization algorithm. During the training process, monitor the performance of the model and adjust the hyperparameters through the validation set to ensure the accuracy and generalization ability of the model. After the training is completed, save the trained state evaluation model for subsequent real-time evaluation.

[0074] S204, inputting the feature data obtained from the current image processing into the state assessment model for real-time assessment, generating assessment results of the worker status and assembly status, and sending the assessment results and related image data to the work island assembly control terminal to manage the assembly quality of the automobile work island, wherein the assessment results include status labels and operation suggestions.

[0075] Integrating the image feature data acquired in real time into the machine learning model enables the system to quickly and accurately evaluate the worker's operating status and assembly status. Through this evaluation, the management side can obtain dynamic information of the production line in real time, so as to make timely adjustments to ensure assembly quality. The status label can clearly identify the current status of the worker, such as "effective operation", "potential risk operation" or "invalid operation", which will further help managers to explore potential safety hazards or quality problems. The operation suggestions are based on the analysis results of the model and provide specific action guidance, such as "please adjust the hand position" or "change tools" to optimize the workflow, improve work safety and product quality. This process realizes visual monitoring of the assembly process, reduces product quality problems caused by human errors, and improves overall production efficiency.

[0076] Specifically, the feature data obtained by the current image processing can be input into the state assessment model for real-time assessment, and a series of forward propagation calculations can be performed to generate real-time assessment results, wherein the assessment results include state labels and operation suggestions. The state labels will clarify the current operating state and assembly state of the worker, and the operation suggestions provide targeted improvement measures, which include adjusting the worker's operating method or changing the use of tools.

[0077] The processed feature data is integrated into a data tensor in a fixed format to ensure that it meets the input requirements of the state assessment model. This involves feature vectorization, which combines image features, posture information, and tool status information into a multidimensional array. The integrated data is input into the state assessment model through the forward propagation algorithm. Each layer of neurons in the model is processed by weighted summation and activation function to convert the input feature data into evaluation results. The softmax function is used to output the state label to determine the worker's operation state and assembly state. In addition, regression analysis is used to obtain operation suggestions, and specific improvement measures are generated based on the output of the model. The output of the model is parsed, the state label and operation suggestions are extracted, and this information is encapsulated into a standardized response format for subsequent data transmission. The evaluation results and related image data are sent to the work island assembly control terminal through the industrial Internet or local area network. This process can be carried out through the RESTful API to ensure that the data is safe and fast during transmission. A feedback mechanism is established at the assembly control terminal. Based on the evaluation results, the system will provide real-time feedback to workers and managers to form a closed-loop system to improve workers' operation awareness and quality management capabilities.

[0078] Through the specific implementation of the above steps, the efficiency, stability and safety of the automobile work island assembly process can be ensured, providing core technical support for the realization of intelligent manufacturing and digital management.

[0079] Furthermore, in practical applications, environmental data of the automobile work island and behavioral data of workers associated with the environment during the assembly process can also be collected to calculate an environmental suitability index that reflects the impact of the environment on assembly.

[0080] In the assembly process of the automobile work island, in order to comprehensively evaluate the impact of the environment on the workers' work efficiency and assembly quality, it is necessary to collect environmental data and behavioral data. This step includes installing sensors and monitoring equipment to collect real-time environmental parameters of the work island, such as light intensity, temperature, humidity, noise level, and carbon dioxide concentration. At the same time, with the help of wearable devices or monitoring systems, record the behavioral data of workers during the operation, including the number of times they wipe sweat, the frequency of drinking water, the pause time for wiping sweat, the pause time for drinking water, etc. The collection of these data can not only reflect the actual operating status of workers in a specific environment, but also provide basic data for the subsequent calculation of the Environmental Suitability Index (EAI).

[0081] For example, a calculation formula for an environmental suitability index can be:

[0082] in, is the weight coefficient of the i-th environmental data, is the suitability score of the i-th environmental data, n is the number of environmental data, is the weight coefficient of the jth worker behavior data associated with the environment, is the impact score of the j-th worker behavior data associated with the environment, and m is the number of the j-th worker behavior data associated with the environment.

[0083] By comprehensively collecting environmental data and worker behavior data, companies can gain an in-depth understanding of the impact of various environmental factors on worker operating efficiency, thereby providing a scientific basis for optimizing working conditions. When the environmental suitability index can systematically reflect the impact of the actual assembly environment on worker performance, managers can promptly identify potential unsuitable factors and take corresponding improvement measures. Secondly, using the environmental suitability index as a quantitative indicator, companies can formulate more reasonable production arrangements, improve overall assembly efficiency, ensure the safety and comfort of the production process, and ultimately improve product quality and employee satisfaction. This process will promote the progress of companies in intelligent manufacturing and digital transformation and enhance market competitiveness.

[0084] It can be seen that at least one image acquisition device is installed at each workstation of the automobile work island, and the image acquisition device can capture the image data of the workers and their working environment in real time; the image processing algorithm is used to process the image data, identify the workers' movements and postures, and extract key features to generate feature data of the workers' assembly; based on the pre-labeled historical feature data, a state assessment model based on machine learning is constructed; the feature data obtained by the current image processing is input into the state assessment model for real-time evaluation, and the evaluation results of the worker status and assembly status are generated, and the evaluation results and related image data are sent to the work island assembly control terminal to control the assembly quality of the automobile work island, thereby being able to integrate image acquisition and advanced data processing technology into an intelligent assembly method, which can provide automobile manufacturers with real-time and accurate worker status monitoring, thereby improving assembly quality and work efficiency, which has important practical significance and market demand.

[0085] Another embodiment of the present invention provides an automobile work island assembly system, see Figure 3 , the system may include:

[0086] The capture module 301 is used to install at least one image acquisition device at each workstation of the automobile work island, and the image acquisition device can capture the image data of the worker and his working environment in real time, and the image data includes the worker's operating actions, working postures, tool usage and the status of the assembled parts;

[0087] An extraction module 302 is used to process the image data using an image processing algorithm, identify the worker's movements and postures, and extract key features to generate feature data of the worker's assembly, wherein the key features include the worker's hand position and the status of the tool;

[0088] A construction module 303 is used to construct a state assessment model based on machine learning based on pre-labeled historical feature data, wherein the model is used to assess the current operation state and assembly state of the worker, wherein the operation state includes effective operation, potential risk operation and invalid operation, and the assembly state includes not started, in progress and completed;

[0089] The evaluation module 304 is used to input the feature data obtained by the current image processing into the state evaluation model for real-time evaluation, generate evaluation results of the worker status and assembly status, and send the evaluation results and related image data to the work island assembly control terminal to manage the assembly quality of the automobile work island, wherein the evaluation results include status labels and operation suggestions.

[0090] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0091] Specifically, in this embodiment, the above storage medium may be configured to store a computer program for performing the following steps:

[0092] S201, installing at least one image acquisition device at each workstation of the automobile work island, wherein the image acquisition device can capture image data of workers and their working environment in real time, wherein the image data includes the workers' operating actions, working postures, tool usage, and status of assembled parts;

[0093] S202, using an image processing algorithm to process the image data, identify the worker's movements and postures, and extract key features to generate feature data of the worker's assembly, wherein the key features include the worker's hand position and the status of the tool;

[0094] S203, based on the pre-labeled historical feature data, construct a state assessment model based on machine learning, wherein the model is used to assess the worker's current operation state and assembly state, wherein the operation state includes effective operation, potential risk operation and invalid operation, and the assembly state includes not started, in progress and completed;

[0095] S204, inputting the feature data obtained from the current image processing into the state assessment model for real-time assessment, generating assessment results of the worker status and assembly status, and sending the assessment results and related image data to the work island assembly control terminal to manage the assembly quality of the automobile work island, wherein the assessment results include status labels and operation suggestions.

[0096] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0097] Specifically, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0098] Specifically, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0099] S201, installing at least one image acquisition device at each workstation of the automobile work island, wherein the image acquisition device can capture image data of workers and their working environment in real time, wherein the image data includes the workers' operating actions, working postures, tool usage, and status of assembled parts;

[0100] S202, using an image processing algorithm to process the image data, identify the worker's movements and postures, and extract key features to generate feature data of the worker's assembly, wherein the key features include the worker's hand position and the status of the tool;

[0101] S203, based on the pre-labeled historical feature data, construct a state assessment model based on machine learning, wherein the model is used to assess the worker's current operation state and assembly state, wherein the operation state includes effective operation, potential risk operation and invalid operation, and the assembly state includes not started, in progress and completed;

[0102] S204, inputting the feature data obtained from the current image processing into the state assessment model for real-time assessment, generating assessment results of the worker status and assembly status, and sending the assessment results and related image data to the work island assembly control terminal to manage the assembly quality of the automobile work island, wherein the assessment results include status labels and operation suggestions.

[0103] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings. The above is only a preferred embodiment of the present invention, but the present invention is not limited to the scope of implementation shown in the drawings. Any changes made according to the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which still do not exceed the spirit covered by the description and drawings, should be within the protection scope of the present invention.

Claims

1. A method for assembling an automobile work island, characterized in that: The method comprises: At least one image acquisition device is installed at each workstation of the automobile work island, and the image acquisition device can capture image data of workers and their working environment in real time, and the image data includes the worker's operating actions, working postures, tool usage and the status of assembled parts; Processing the image data using an image processing algorithm to identify the worker's movements and postures and extract key features to generate feature data of the worker's assembly, wherein the key features include the worker's hand position and the status of the tool; Based on the pre-labeled historical feature data, a state assessment model based on machine learning is constructed, and the model is used to assess the current operating state and assembly state of the worker, wherein the operating state includes effective operation, potential risk operation and invalid operation, and the assembly state includes not started, in progress and completed; Based on the pre-labeled historical feature data, a state assessment model based on machine learning is constructed, and the model is used to assess the current operating state and assembly state of the worker, including: Collect historical image datasets from the assembly process of automobile work islands, which cover various operations and assembly situations. The data sources include operations of different workstations and different workers, as well as the use of various tools and parts. Extract the historical feature dataset of workers' assembly from the historical image set, and use support vector machine, random forest or deep learning model to train the state assessment model to obtain a trained state assessment model. In this model, a multi-task learning framework is used to simultaneously train the workers' operation state and assembly state, and mutual learning between different tasks is promoted by sharing the underlying feature representation. The feature data obtained by the current image processing is input into the state assessment model for real-time assessment, and the assessment results of the worker status and assembly status are generated. The assessment results and related image data are sent to the work island assembly control terminal to manage the assembly quality of the automobile work island, wherein the assessment results include status labels and operation suggestions.

2. The method according to claim 1, characterized in that The image data is processed by an image processing algorithm to identify the worker's movements and postures and extract key features to generate feature data of the worker's assembly, including: Using an accelerated robust feature algorithm to detect key feature points in the image, applying an ORB algorithm to describe the key feature points, and generating feature descriptors, wherein the key feature points include the worker's body feature points and tool feature points; The extracted key feature points and their feature descriptors are input into the pre-trained motion classification model to identify the worker's hand movements. At the same time, the posture information of the worker's specific joints is obtained by combining the posture estimation algorithm. The posture information and feature descriptors are used together to analyze whether the worker's posture meets the optimal process standard. According to the tool status, feature descriptors are used to realize tool feature comparison, wherein the tool outline is identified by feature matching algorithm, and the generated feature descriptors are used for matching to determine whether the tool is in the correct use state, incorrect use state or standby state; The extracted hand movements, posture information, and tool status are integrated into a structured feature set to obtain the feature data of the worker's assembly.

3. The method according to claim 2, characterized in that The feature data obtained by the current image processing is input into the state evaluation model for real-time evaluation, and the evaluation results of the worker state and the assembly state are generated. The evaluation results and related image data are sent to the work island assembly control terminal to control the assembly quality of the automobile work island, including: The feature data obtained by the current image processing is input into the state assessment model for real-time assessment, and a series of forward propagation calculations are performed to generate real-time assessment results, wherein the assessment results include state labels and operation suggestions. The state labels will clarify the current operation state and assembly state of the worker, and the operation suggestions provide targeted improvement measures, which include adjusting the worker's operation method or changing the use of tools.

4. The method according to claim 3, characterized in that: The method further comprises: The environmental data of the automobile work island and the behavioral data of workers related to the environment during the assembly process are collected to calculate the environmental suitability index used to reflect the impact of the environment on assembly.

5. An automobile work island assembly system, characterized in that: The system comprises: A capture module is used to install at least one image acquisition device at each workstation of the automobile work island, and the image acquisition device can capture image data of workers and their working environment in real time, and the image data includes the worker's operating actions, working postures, tool usage and the status of assembled parts; An extraction module, used to process the image data using an image processing algorithm, identify the worker's movements and postures, and extract key features to generate feature data of the worker's assembly, wherein the key features include the worker's hand position and the status of the tool; A construction module is used to construct a state assessment model based on machine learning based on pre-labeled historical feature data, and the model is used to assess the current operation state and assembly state of the worker, wherein the operation state includes effective operation, potential risk operation and invalid operation, and the assembly state includes not started, in progress and completed; the state assessment model based on machine learning based on pre-labeled historical feature data is constructed, and the model is used to assess the current operation state and assembly state of the worker, including: Collect historical image datasets from the assembly process of automobile work islands, which cover various operations and assembly situations. The data sources include operations of different workstations and different workers, as well as the use of various tools and parts. Extract the historical feature dataset of workers' assembly from the historical image set, and use support vector machine, random forest or deep learning model to train the state assessment model to obtain a trained state assessment model. In this model, a multi-task learning framework is used to simultaneously train the workers' operation state and assembly state, and mutual learning between different tasks is promoted by sharing the underlying feature representation. The evaluation module is used to input the feature data obtained by the current image processing into the state evaluation model for real-time evaluation, generate evaluation results of the worker status and assembly status, and send the evaluation results and related image data to the work island assembly control terminal to manage the assembly quality of the automobile work island, wherein the evaluation results include status labels and operation suggestions.

6. The system according to claim 5, characterized in that The extraction module is specifically used for: Using an accelerated robust feature algorithm to detect key feature points in the image, applying an ORB algorithm to describe the key feature points, and generating feature descriptors, wherein the key feature points include the worker's body feature points and tool feature points; The extracted key feature points and their feature descriptors are input into the pre-trained motion classification model to identify the worker's hand movements. At the same time, the posture information of the worker's specific joints is obtained by combining the posture estimation algorithm. The posture information and feature descriptors are used together to analyze whether the worker's posture meets the optimal process standard. According to the tool status, feature descriptors are used to realize tool feature comparison, wherein the tool outline is identified by feature matching algorithm, and the generated feature descriptors are used for matching to determine whether the tool is in the correct use state, incorrect use state or standby state; The extracted hand movements, posture information, and tool status are integrated into a structured feature set to obtain the feature data of the worker's assembly.

7. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 4 when executed.

8. An electronic device, comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 4.

Images