Process tracking method, device and equipment based on equipment maintenance, and storage medium

Abstract

The application discloses a device maintenance-based process tracking method and device, equipment and a storage medium, wherein the method comprises the following steps: acquiring personnel-related data of a maintenance personnel in a maintenance scene, wherein the personnel-related data comprises a posture category of the maintenance personnel and a first spatial relative position where the maintenance personnel is located; acquiring component-related data in the maintenance scene, wherein the component-related data comprises a component category and a second spatial relative position where a component corresponding to the component category is located; determining local behavior information of the maintenance personnel on the component according to the personnel-related data and the component-related data; and performing maintenance process tracking on the maintenance scene based on the local behavior information. The whole-process monitoring of the maintenance process is realized, the work supervision on the maintenance personnel is improved, and the safe operation of the equipment is better guaranteed.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a process tracking method based on equipment maintenance, a process tracking device based on equipment maintenance, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Equipment maintenance (or equipment service) is a technical management measure implemented according to a pre-defined plan or relevant technical specifications to prevent equipment performance degradation or reduce the probability of equipment failure. It combines repair and upkeep. Equipment maintenance is the foundation for normal equipment operation and an effective measure to prevent accidents.

[0003] Currently, the common maintenance model for equipment involves maintenance companies scheduling maintenance personnel to perform maintenance or repair operations based on the maintenance cycle or repair notice for each piece of equipment. However, the lack of monitoring of the work process of maintenance personnel can easily lead to problems such as irresponsible maintenance, untimely emergency repairs, and poor completion of repair procedures. Summary of the Invention

[0004] This application provides a process tracking method, apparatus, equipment, and storage medium for equipment maintenance, in order to solve the supervision problems that easily arise in the existing equipment maintenance process due to the lack of monitoring, such as irresponsible maintenance, untimely emergency repairs, and poor completion of maintenance procedures.

[0005] According to a first aspect of this application, a process tracking method based on equipment maintenance is provided, the method comprising:

[0006] Acquire personnel-related data of maintenance personnel in the maintenance scenario, including the posture category of the maintenance personnel and the relative position of the maintenance personnel in a first space;

[0007] Acquire component-related data in the maintenance scenario, including component categories and the relative second spatial position of the components corresponding to those component categories;

[0008] Based on the personnel-related data and the component-related data, determine the local behavior information of the maintenance personnel towards the component;

[0009] Based on the local behavior information, the maintenance process is tracked for the maintenance scenario.

[0010] According to a second aspect of this application, a process tracking device based on equipment maintenance is provided, the device comprising:

[0011] The personnel data acquisition module is used to acquire personnel-related data of maintenance personnel in the maintenance scenario. The personnel-related data includes the posture category of the maintenance personnel and the relative position of the maintenance personnel in the first space.

[0012] The component data acquisition module is used to acquire component-related data in the maintenance scenario. The component-related data includes the component category and the second spatial relative position of the component corresponding to the component category.

[0013] The behavior information determination module is used to determine the local behavior information of the maintenance personnel towards the component based on the personnel-related data and the component-related data;

[0014] The process tracking module is used to track the maintenance process of the maintenance scenario based on the local behavior information.

[0015] According to a third aspect of this application, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method described in any embodiment of this application.

[0019] According to a fourth aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the method described in any embodiment of this application.

[0020] In this embodiment, by acquiring personnel-related data of maintenance personnel and component-related data of the components to be maintained in the maintenance scenario, the local behavioral information of the maintenance personnel on the components is automatically determined, and then the maintenance process is tracked based on this local behavioral information. The entire process uses continuous data in the time and spatial domains to achieve full-process monitoring of the maintenance process, improving the supervision of maintenance personnel's work and thus better ensuring the safe operation of the equipment.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a process tracking method based on equipment maintenance provided in Embodiment 1 of this application;

[0024] Figure 2 This is a schematic diagram of a maintenance scenario provided in Embodiment 1 of this application;

[0025] Figure 3 This is a schematic diagram of the structure of a process tracking device based on equipment maintenance provided in Embodiment 2 of this application;

[0026] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Example 1

[0030] Figure 1This is a flowchart of a process tracking method based on equipment maintenance provided in Embodiment 1 of this application. This embodiment can be applied to a process tracking device based on equipment maintenance. The device can be set in a server, management backend, or a local device with a processor, etc. This embodiment does not limit the device.

[0031] This embodiment can be applied to scenarios where maintenance personnel are tracking and monitoring the equipment maintenance process in a maintenance environment. However, this embodiment does not limit the maintenance scenario; any scenario requiring maintenance personnel to perform regular maintenance, inspection, or repair of equipment falls within the protection scope of this embodiment. For example, a maintenance scenario may include elevator maintenance, in which case the corresponding equipment is the elevator equipment, and the corresponding components are the components included in the elevator equipment during operation. Similarly, a maintenance scenario may include fire equipment maintenance, in which case the corresponding equipment is the fire equipment, and the corresponding components are the components included in the fire equipment.

[0032] like Figure 1 As shown, the method may include the following steps:

[0033] Step 110: Obtain personnel-related data for maintenance personnel in the maintenance scenario.

[0034] This step primarily involves acquiring personnel-related data for maintenance personnel within the maintenance scenario. This personnel-related data can include data generated by maintenance personnel within the maintenance scenario. Once recorded, this data can be used for subsequent process tracking during maintenance. For example, personnel-related data may include, but is not limited to: the posture category of the maintenance personnel, their relative position in a first spatial location, and relevant data collected by sensors set up within the maintenance scenario.

[0035] In one embodiment, if the personnel-related data includes posture categories, then step 110 may further include the following steps:

[0036] Step 110-1: Obtain the first image data sequence containing the maintenance personnel.

[0037] In implementation, the first image data sequence can be acquired by a first acquisition device positioned within the maintenance scenario. This first acquisition device can be mobile, allowing maintenance personnel to locate a suitable installation site upon arrival at the maintenance location and then install the device there. This installation location enables the first acquisition device to capture information from the entire maintenance scenario, at least capturing the maintenance operations performed by the personnel. The number of maintenance personnel can be one or more.

[0038] In one implementation, the first acquisition device may include multiple sensors, exemplarily including image sensors, depth sensors, triaxial sensors, electronic compasses, etc. Then, each first image data in the first image data sequence may be image data acquired by the image sensor. The first image data may at least include maintenance personnel, and in other examples, may also include part or all of the component being maintained by the maintenance personnel.

[0039] Step 110-3: Extract the first area image where the maintenance personnel are located from each first image data.

[0040] In one implementation, a pre-trained human target detection network can be used to detect the bounding box of the maintenance personnel from the first image. This bounding box can be represented by positional information (x1, y1, w1, h1), where (x1, y1) are the coordinates of the top-left corner of the bounding box, and (w1, h1) are the width and height of the bounding box, respectively. After obtaining the bounding boxes for each first image data, these bounding boxes can be extracted as the first region image representing the maintenance personnel. During extraction, the image can be cropped using the bounding box as the boundary, or the cropping can be performed by extending the bounding box outwards by a preset pixel distance.

[0041] In this embodiment, the target detection algorithm used in the human target detection network is not limited.

[0042] Step 110-5: Perform pose recognition on the continuous first region image sequence to determine the pose category of the maintenance personnel.

[0043] After obtaining the first region images corresponding to each first image data, the first region images in a continuous time series will form a first region image sequence, and then pose recognition will be performed on the first region image sequence. In one implementation, a pre-generated human pose estimation network can be used to perform pose recognition on the first region image sequence to obtain the pose category of the maintenance personnel. The pose category may include, but is not limited to, standing, lying down, semi-reclining, bending over, reaching out, and pulling.

[0044] In one implementation, the human pose estimation network can be generated as follows: A human pose dataset of the current maintenance scene is collected to train the Alphapose pose estimation network. Through training with the dataset, the Alphapose network learns the heatmap distribution of human joints and obtains the numerical coordinates of the joints within the human bounding box through heatmap regression. These coordinates are then mapped back to the input image, finally yielding the coordinates of the human joints in the pixel coordinate system of the input image. Human joints may include, for example, the nose, neck, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, spine, left hip, left knee, left ankle, right hip, right knee, and right ankle. Then, a 3D human pose reconstruction network is used to reconstruct the 2D human pose of the first region image sequence into 3D numerical coordinates of the human joints in the image sequence. Next, based on the 3D human pose dataset Human3.6M and the 2D human pose as input, the 3D human pose reconstruction model 3D-baseline is trained. The obtained 2D human position coordinates are then used as input to the model to output 3D human position coordinates. In other words, the input is the estimated 2D coordinates of the 15 joints mentioned above, and the output is the 3D coordinates of the 15 joints. Then, the spatiotemporal relationship of human posture is obtained from the sequence of 3D numerical coordinates of human joints using the GCN network model, so as to realize the recognition of human posture in the current maintenance environment; based on the 3D human posture behavior dataset Human3.6M, the human posture estimation model is trained using 3D human posture as input.

[0045] In another embodiment, if the personnel-related data includes a first spatial relative position, which is the spatial relative position between the maintenance personnel and the first data acquisition device, it may, for example, include the distance and angle between the maintenance personnel and the first data acquisition device. In implementation, a visual positioning algorithm can be used to calculate the first spatial relative position. Specifically, step 110 may further include the following steps:

[0046] Step 110-2: Determine the key points of the maintenance personnel from the first area image.

[0047] In one implementation, the first region image can be input into the human pose estimation network described above, and the key points of the maintenance personnel output by the network can be obtained. These key points can include key points that can reflect human pose, such as head key points, limb key points, torso key points, etc.

[0048] Step 110-4: Obtain the first depth image acquired by the first acquisition device, and perform pixel matching between the first region image and the first depth image to determine the three-dimensional coordinates of each key point in the camera coordinate system.

[0049] In one implementation, the depth information of each keypoint can be determined from the first depth image by pixel matching. Assuming the pixel coordinates of each keypoint in the first region image are (u, v), combined with the focal length f of the first acquisition device and the principal point coordinates (c... x ,c y This allows us to calculate the three-dimensional coordinates of each key point in the camera coordinate system.

[0050] In one implementation, the three-dimensional coordinates (X, Y, Z) of each key point in the camera coordinate system can be calculated using the following formula. c ,Y c Z c ):

[0051] Z c =depth

[0052]

[0053]

[0054] Step 110-6: Obtain the first angle information of the first acquisition device, and based on the first angle information, convert the three-dimensional coordinates of each key point in the camera coordinate system into three-dimensional coordinates in the world coordinate system, wherein the world coordinate system is constructed with the position of the first acquisition device projected onto the ground as the origin.

[0055] For example, the first angle information may include at least an attitude angle and a direction angle. In implementation, the attitude angle of the first acquisition device can be obtained by a three-axis sensor on the first acquisition device, and the direction angle of the first acquisition device can be obtained by an electronic compass on the first acquisition device.

[0056] In implementation, a coordinate transformation matrix can be constructed based on the first angle information, and the three-dimensional coordinates (X, Y, F, Z) of each key point in the camera coordinate system can be determined based on this coordinate transformation matrix. c ,Y c Z c Convert to three-dimensional coordinates in the world coordinate system (X) w ,Y w Z w ).

[0057] In this world coordinate system, the origin is the position projected onto the ground by the first data acquisition device; the z-axis is perpendicular to the ground and points towards the sky; and the x-axis and y-axis lie on the ground. Then Z... w It can represent the height of key points above the ground.

[0058] This embodiment does not limit the method of constructing the coordinate transformation matrix using the first angle information; it can be any existing construction method.

[0059] Step 110-8: Calculate the distance and angle between each key point and the first acquisition device based on the three-dimensional coordinates of each key point in the world coordinate system, and use this as the first spatial relative position.

[0060] After determining the three-dimensional coordinates of each key point in the world coordinate system, the three-dimensional coordinates of the key points of the first acquisition device (e.g., the midpoint between the image sensor and the depth sensor) can also be determined based on the world coordinate system. Then, the distance between each key point of the maintenance personnel and the first acquisition device is calculated using a distance calculation formula, and the angle between each key point of the maintenance personnel and the first acquisition device is calculated using an angle calculation formula. Finally, these distances and angles are used as the first spatial relative positions.

[0061] In addition, the height information (the z-axis value in the world coordinate system) of each key point can be used as the first spatial relative position.

[0062] Step 120: Obtain component-related data in the maintenance scenario.

[0063] This step primarily involves acquiring component-related data for the parts in the maintenance scenario. This data can include information related to the components of each device within the maintenance scenario. Once recorded, this data can be used for subsequent process tracking during maintenance. For example, the component-related data may include, but is not limited to: component category, the relative location of the component within that category in a second space, and relevant data collected by sensors set up in the maintenance scenario.

[0064] In one embodiment, if the component-related data includes component categories, step 120 may further include the following steps:

[0065] Step 120-1: Obtain a second image data sequence containing the component.

[0066] In implementation, the second image data sequence can be acquired by a second acquisition device positioned in the maintenance scenario. This second acquisition device can be a wearable device, worn by maintenance personnel upon arrival at the maintenance site to acquire images of the component. The wearing position allows the second acquisition device to capture complete information about the component. For example, such as... Figure 2 As shown, the second data acquisition device 20 can be worn on the head of maintenance personnel, 10 is the first data acquisition device, and 30 is a component.

[0067] In one implementation, the second acquisition device may also include multiple sensors, such as an image sensor, a depth sensor, a triaxial sensor, an electronic compass, etc. Each second image data point in the second image data sequence can be image data acquired by the image sensor, and the second image data can include at least the component to be maintained, preferably a component currently being maintained.

[0068] Step 120-3: Extract the second region image where the component is located from each second image data.

[0069] In one implementation, a pre-trained part object detection network can be used to detect the bounding boxes of the parts from the second image data. These bounding boxes can be represented by positional information (x2, y2, w2, h2), where (x2, y2) are the coordinates of the top-left corner of the bounding box, and (w2, h2) are the width and height of the bounding box, respectively. After obtaining one or more bounding boxes from each of the second image data sets, each bounding box can be extracted as a second region image representing the part. During extraction, the image can be cropped using the bounding box as the boundary, or the cropping can be performed by extending the bounding box outwards by a predetermined pixel distance.

[0070] In this embodiment, the target detection algorithm used in the component target detection network is not limited.

[0071] In actual processing, the number of components in the second image may exceed one. In this case, it is necessary to determine which component needs to be processed, thereby selecting the second image region corresponding to the component to be processed. There are various methods for selection, including but not limited to: filtering out incomplete components based on component integrity detection; obtaining the depth information of each component and determining the component located in the foreground as the current component to be processed; or selecting the component with the highest frequency of occurrence in consecutive image frames as the current component to be processed.

[0072] Step 120-5 involves performing component identification on the continuous second region image sequence to determine the component category. After obtaining the second region image corresponding to the current component to be processed in each second image data, the continuous time series of second region images will form a second region image sequence, and then component category identification will be performed on this second region image sequence. In one implementation, a pre-generated component category identification network can be used to perform component category identification on the second region image sequence to determine what type of component the current component is. For example, in an elevator maintenance scenario, component categories may include, but are not limited to: wire rope, brake, speed governor, etc.

[0073] This embodiment does not limit the training method of the component category recognition network. In one implementation, the sample set used for training the component category recognition network can be all the components appearing in the current maintenance scenario. By labeling each component sample with its corresponding component category in advance and extracting the feature point information of each component sample, and training it using a preset classification algorithm, the component category recognition network can be obtained. In another embodiment, the component-related data may also include the component status, which may include, for example, working status, component structural integrity, etc. The working status describes whether the component is currently in a working state or a non-working state, or which working mode the component is in. The component structural integrity describes whether the current component has sub-components, i.e., whether any sub-components are missing. Then step 120 may include the following steps:

[0074] Component state detection is performed on the second region image sequence to determine the component state.

[0075] In one implementation, component state detection logic can be pre-defined, and the current component's state can be detected according to this logic to determine its state. For example, if the component state is "working," images of the component in its working state and its non-working state can be queried based on its component category. Then, the currently acquired second region image is compared with these images to determine whether the component in the second region image is in a working or non-working state. As another example, if the component state is "component structural integrity," images of the component with complete structure can be queried based on its component category. Then, the currently acquired second region image is compared with these images. If the components in both images match, the current component is complete; otherwise, the current component is incomplete.

[0076] In another embodiment, the component-related data may further include component quality, which includes whether defects exist on the component surface. Then step 120 may include the following step: performing component quality detection on the second region image sequence to determine the component quality.

[0077] In one implementation, component quality inspection logic can be pre-defined, and the current component can be inspected according to this logic to determine its quality. For example, the component quality inspection logic can be a component quality inspection model. Pre-collected surface defects that are prone to occur in various component categories within the current maintenance scenario can be used as sample data to train the quality inspection model. Then, each second region image in the second region image sequence is input into this quality inspection model, which detects whether the image contains a corresponding defect and outputs a quality result. This quality result can, for example, use the defect category as the outcome.

[0078] In another embodiment, if the component-related data includes a second spatial relative position, then step 120 may include the following steps:

[0079] Step 120-2: Determine the key points of the component from the second region image.

[0080] The key points of a component can include pixels representing its shape or contour features. In implementation, a component key point extraction model can be used to determine the key points of the component from the second region image.

[0081] Step 120-4: Acquire the second depth image acquired by the second acquisition device, and perform pixel matching between the second region image and the second depth image to determine the three-dimensional coordinates of each key point of the component in the camera coordinate system.

[0082] Step 120-6: Obtain the second angle information of the second acquisition device, and based on the second angle information, convert the three-dimensional coordinates of each key point of the component in the camera coordinate system into three-dimensional coordinates in the world coordinate system, wherein the world coordinate system is constructed with the position of the second acquisition device projected onto the ground as the origin.

[0083] Step 120-8: Based on the three-dimensional coordinates of each key point of the component in the world coordinate system, calculate the distance and angle between each key point and the second acquisition device, as the second spatial relative position.

[0084] The second spatial relative position refers to the relative position of the component and the second acquisition device in the scene. The calculation method for the second spatial relative position is similar to that for the first spatial relative position, and can be referred to the method for the first spatial relative position described above. This embodiment will not repeat the details here.

[0085] Step 130: Based on the personnel-related data and the component-related data, determine the local behavior information of the maintenance personnel towards the component.

[0086] Local behavior information can be the actions of maintenance personnel in maintaining a certain component during a certain period of time. For example, in the elevator maintenance scenario, local behavior information may include, but is not limited to: measuring component dimensions, tightening screws, pressing buttons, measuring electrical properties, etc.

[0087] In one embodiment, step 130 may further include the following steps:

[0088] Step 130-1: Determine the relative positional relationship between the maintenance personnel and the component based on the first spatial relative position and the second spatial relative position.

[0089] Specifically, the second data acquisition device is worn by the maintenance personnel. Once the angle and distance between the component and the second data acquisition device are determined based on the relative position in the second space, this angle and distance can be used as the angle and distance between the component and the maintenance personnel.

[0090] In addition, the first spatial relative position may include the ground clearance of key points for maintenance personnel. The second spatial relative position includes the ground clearance of key points for components. By comparing the maximum ground clearance of the maintenance personnel with the current maximum ground clearance of the component, and by comparing the minimum ground clearance of the maintenance personnel with the current minimum ground clearance of the component, the vertical position of the component relative to the maintenance personnel can be determined, for example, whether the component is above or below the maintenance personnel.

[0091] The distance, angle, and vertical orientation between the aforementioned components and maintenance personnel can all be referred to as the relative positional relationship between the maintenance personnel and the components. Step 130-2: Generate matching information based on the posture category, the component category, and the relative positional relationship, and match the matching information with multiple pre-generated local behavior templates, so that the successfully matched local behavior templates are used as the local behavior information of the matching information. Each local behavior template includes a template posture category, a template component category, and a template relative positional relationship.

[0092] Once information such as the posture category of the maintenance personnel, the component category of the components operated by the maintenance personnel, and the relative positional relationship between the maintenance personnel and the components are obtained within a certain time period, this information can be used as the matching information. This matching information is then quickly matched against multiple pre-generated local behavior templates, and the successfully matched local behavior template is taken as the local behavior information of the current matching information. Specifically, the local behavior template may also include the template posture category, template component category, and template relative positional relationship. When a local behavior template exists whose template posture category is the same as the current posture category, whose template component category is the same as the current component category, and which has the same or similar relative positional relationship, then this local behavior template can be determined as a successfully matched local behavior template.

[0093] Step 140: Based on the local behavior information, perform maintenance process tracking for the maintenance scenario.

[0094] Once local behavior information is obtained, it can be recorded. This allows for subsequent tracking of the maintenance process based on this local behavior information.

[0095] In one embodiment, step 140 may further include the following steps:

[0096] The system acquires information on one or more local actions performed by the maintenance personnel in the maintenance scenario, and organizes this information into a local action list in chronological order. The local action list is then matched with one or more previously generated process templates, and the successfully matched process template is used as the process category corresponding to the local action list. For example, in an elevator maintenance scenario, the process template may include insulation resistance testing, brake testing, and wire rope testing.

[0097] In this step, multiple process templates can be pre-set, each containing multiple process steps. Then, the previously recorded local behavior information in the current maintenance scenario is used as steps to form a local behavior list, i.e., a step list. This step list is compared with the order of steps in each process template. If a process template exists whose step order is the same as or similar to the step order in the current local behavior list, then the most similar or completely identical process template is taken as the matching process template, and the process category corresponding to this matching process template is taken as the process category corresponding to the current local behavior list.

[0098] In one embodiment, after determining the process category, an integrity check can be performed on the current process. Step 140 may further include the following steps:

[0099] Based on the list of local behaviors, determine whether all process steps in the successfully matched process template have been completed; if so, determine that the current process category is completely completed, meaning the current process is fully completed. If not, determine that the current process category is not completed. Based on the above judgment conditions, a prompt message can also be issued to the user.

[0100] In this embodiment, by acquiring personnel-related data of maintenance personnel and component-related data of the components to be maintained in the maintenance scenario, the local behavioral information of the maintenance personnel on the components is automatically determined, and then the maintenance process is tracked based on this local behavioral information. The entire process uses continuous data in the time and spatial domains to achieve full-process monitoring of the maintenance process, improving the supervision of maintenance personnel's work and thus better ensuring the safe operation of the equipment.

[0101] Example 2

[0102] Figure 3 A schematic diagram of a process tracking device based on equipment maintenance provided in Embodiment 2 of this application may include the following modules:

[0103] The personnel data acquisition module 210 is used to acquire personnel-related data of maintenance personnel in the maintenance scenario. The personnel-related data includes the posture category of the maintenance personnel and the relative position of the maintenance personnel in the first space.

[0104] The component data acquisition module 220 is used to acquire component-related data in the maintenance scenario. The component-related data includes the component category and the second spatial relative position of the component corresponding to the component category.

[0105] The behavior information determination module 230 is used to determine the local behavior information of the maintenance personnel on the component based on the personnel-related data and the component-related data;

[0106] The process tracking module 240 is used to track the maintenance process of the maintenance scenario based on the local behavior information.

[0107] In one embodiment, the personnel data acquisition module 210 is specifically used for:

[0108] Obtain a first image data sequence containing the maintenance personnel;

[0109] Extract the image of the first area where the maintenance personnel are located from each of the first image data;

[0110] Pose recognition is performed on a continuous sequence of images of the first region to determine the pose category of the maintenance personnel.

[0111] In another embodiment, the first image data is acquired by a first acquisition device located in the maintenance scenario; the personnel data acquisition module 210 is further configured to:

[0112] Identify the key points of the maintenance personnel from the first area image;

[0113] Acquire a first depth image captured by the first acquisition device, and perform pixel matching between the first region image and the first depth image to determine the three-dimensional coordinates of each key point in the camera coordinate system;

[0114] The first angle information of the first acquisition device is obtained, and based on the first angle information, the three-dimensional coordinates of each key point in the camera coordinate system are converted into three-dimensional coordinates in the world coordinate system, wherein the world coordinate system is constructed with the position of the first acquisition device projected onto the ground as the origin.

[0115] Based on the three-dimensional coordinates of each key point in the world coordinate system, the distance and angle between each key point and the first acquisition device are calculated as the first spatial relative position.

[0116] In one embodiment, the component data acquisition module 220 is specifically used for:

[0117] Obtain a second image data sequence containing the component;

[0118] Extract the image of the second region where the component is located from each of the second image data;

[0119] Component identification is performed on a continuous sequence of second region images to determine the component category.

[0120] In another embodiment, the component-related data further includes component status, and the component data acquisition module 220 is further configured to:

[0121] Component state detection is performed on the second region image sequence to determine the component state, which includes the working state.

[0122] In another embodiment, the component-related data further includes component quality, and the component data acquisition module 220 is further configured to:

[0123] The second region image sequence is subjected to component quality detection to determine the component quality, which includes whether there are defects on the component surface.

[0124] In another embodiment, the second image data is acquired by a second acquisition device located in the maintenance scenario; the component data acquisition module 220 is further configured to:

[0125] Determine the key points of the component from the second region image;

[0126] The second depth image acquired by the second acquisition device is obtained, and the second region image and the second depth image are pixel matched to determine the three-dimensional coordinates of each key point of the component in the camera coordinate system;

[0127] The second angle information of the second acquisition device is obtained, and based on the second angle information, the three-dimensional coordinates of each key point of the component in the camera coordinate system are converted into three-dimensional coordinates in the world coordinate system, wherein the world coordinate system is constructed with the position of the second acquisition device projected onto the ground as the origin;

[0128] Based on the three-dimensional coordinates of each key point of the component in the world coordinate system, the distance and angle between each key point and the second acquisition device are calculated as the second spatial relative position.

[0129] In one embodiment, the behavior information determination module 230 may include the following sub-modules:

[0130] The relative position relationship determination submodule is used to determine the relative position relationship between the maintenance personnel and the component based on the first spatial relative position and the second spatial relative position;

[0131] The local behavior template matching submodule is used to generate matching information based on the posture category, the component category and the relative position relationship, and to match the matching information with a plurality of pre-generated local behavior templates, so as to use the successfully matched local behavior template as the local behavior information of the matching information. Each local behavior template includes a template posture category, a template component category and a template relative position relationship.

[0132] In one embodiment, the first spatial relative position includes the distance and angle between the first data acquisition device disposed in the maintenance scene and the maintenance personnel; the second spatial relative position includes the angle and distance between the second data acquisition device disposed on the maintenance personnel and the component.

[0133] The relative position relationship determination submodule is specifically used for:

[0134] The distance and angle between the component and the second acquisition device are taken as the distance and angle between the component and the maintenance personnel.

[0135] In one embodiment, the process tracking module 240 is specifically used for:

[0136] Obtain information on one or more local behaviors performed by the maintenance personnel in the maintenance scenario, and organize the information on one or more local behaviors into a list of local behaviors in chronological order;

[0137] The local behavior list is matched with one or more previously generated process templates, and the successfully matched process template is taken as the process category corresponding to the local behavior list.

[0138] In another embodiment, the process tracking module 240 is further configured to:

[0139] Based on the list of local behaviors, determine whether all process steps in the successfully matched process template have been completed;

[0140] If so, then the current process category is considered complete.

[0141] If not, then the current process category is determined to be incomplete.

[0142] The process tracking device based on equipment maintenance provided in this application can execute the process tracking method based on equipment maintenance provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the execution method.

[0143] Example 3

[0144] Figure 4A schematic diagram of the structure of an electronic device 10 that can be used to implement embodiments of the methods of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0145] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0146] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0147] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as those described in Embodiment 1 or Embodiment 2.

[0148] In some embodiments, the methods described in Embodiment 1 or Embodiment 2 may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the methods described in Embodiment 1 or Embodiment 2 above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the methods described in Embodiment 1 or Embodiment 2 by any other suitable means (e.g., by means of firmware).

[0149] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0150] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0151] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0153] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0154] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0155] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0156] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A process tracking method based on equipment maintenance, characterized in that, The method includes: Acquire personnel-related data of maintenance personnel in the maintenance scenario. The personnel-related data includes the posture category of the maintenance personnel and the first spatial relative position of the maintenance personnel. The first spatial relative position is the spatial relative position of the maintenance personnel and the first acquisition device. Acquire component-related data in the maintenance scenario. The component-related data includes component categories and the second spatial relative position of the component corresponding to the component category. The second spatial relative position is the relative position of the component and the second acquisition device in the scenario. Based on the personnel-related data and the component-related data, determine the local behavior information of the maintenance personnel towards the component; Based on the local behavior information, the maintenance process of the maintenance scenario is tracked. The step of determining the maintenance personnel's local behavior information on the component based on the personnel-related data and the component-related data includes: The relative positional relationship between the maintenance personnel and the component is determined based on the first spatial relative position and the second spatial relative position; Based on the posture category, the component category, and the relative positional relationship, matching information is generated, and the matching information is matched with multiple pre-generated local behavior templates. The successfully matched local behavior templates are used as the local behavior information of the matching information. Each local behavior template includes a template posture category, a template component category, and a template relative positional relationship. The step of tracking the maintenance process of the maintenance scenario based on the local behavior information includes: Obtain information on one or more local behaviors performed by the maintenance personnel in the maintenance scenario, and organize the information on one or more local behaviors into a list of local behaviors in chronological order; The step sequence of the local behavior list is matched with the step sequence of one or more previously generated process templates, and the successfully matched process template is taken as the process category corresponding to the local behavior list. The acquisition of component-related data in the maintenance scenario includes: Obtain a second image data sequence containing the component; Extract the image of the second region where the component is located from each of the second image data; Component identification is performed on a continuous second region image sequence to determine the component category; wherein, a pre-generated component category identification network is used to identify the component category of the second region image sequence; The first spatial relative position includes the distance and angle between the first data acquisition device positioned in the maintenance scenario and the maintenance personnel; the second spatial relative position includes the angle and distance between the second data acquisition device positioned on the maintenance personnel and the component. Determining the relative positional relationship between the maintenance personnel and the component based on the first spatial relative position and the second spatial relative position includes: The distance and angle between the component and the second acquisition device are taken as the distance and angle between the component and the maintenance personnel. The process template includes multiple process steps, and the step of tracking the maintenance process of the maintenance scenario based on the local behavior information further includes: Based on the list of local behaviors, determine whether all process steps in the successfully matched process template have been completed; If so, then the current process category is determined to be complete; If not, then the current process category is determined to be incomplete.

2. The method according to claim 1, characterized in that, The acquisition of personnel-related data of maintenance personnel in the maintenance scenario includes: Obtain a first image data sequence containing the maintenance personnel; Extract the image of the first area where the maintenance personnel are located from each of the first image data; Pose recognition is performed on a continuous sequence of images of the first region to determine the pose category of the maintenance personnel.

3. The method according to claim 2, characterized in that, The first image data is acquired by a first acquisition device located in the maintenance scenario; The acquisition of personnel-related data of maintenance personnel in the maintenance scenario also includes: Identify the key points of the maintenance personnel from the first area image; Acquire a first depth image captured by the first acquisition device, and perform pixel matching between the first region image and the first depth image to determine the three-dimensional coordinates of each key point in the camera coordinate system; The first angle information of the first acquisition device is obtained, and based on the first angle information, the three-dimensional coordinates of each key point in the camera coordinate system are converted into three-dimensional coordinates in the world coordinate system, wherein the world coordinate system is constructed with the position of the first acquisition device projected onto the ground as the origin. Based on the three-dimensional coordinates of each key point in the world coordinate system, the distance and angle between each key point and the first acquisition device are calculated as the first spatial relative position.

4. The method according to claim 1, characterized in that, The component-related data also includes component status. The process of obtaining the component-related data in the maintenance scenario further includes: Component state detection is performed on the second region image sequence to determine the component state, which includes the working state.

5. The method according to claim 1, characterized in that, The component-related data also includes component quality, and the process of obtaining the component-related data in the maintenance scenario further includes: The second region image sequence is subjected to component quality detection to determine the component quality, which includes whether there are defects on the component surface.

6. The method according to any one of claims 1-5, characterized in that, The second image data is acquired by a second acquisition device located in the maintenance scenario; The process of acquiring component-related data in the maintenance scenario also includes: Determine the key points of the component from the second region image; Acquire a second depth image captured by the second acquisition device, and perform pixel matching between the second region image and the second depth image to determine the three-dimensional coordinates of each key point of the component in the camera coordinate system; The second angle information of the second acquisition device is obtained, and based on the second angle information, the three-dimensional coordinates of each key point of the component in the camera coordinate system are converted into three-dimensional coordinates in the world coordinate system, wherein the world coordinate system is constructed with the position of the second acquisition device projected onto the ground as the origin; Based on the three-dimensional coordinates of each key point of the component in the world coordinate system, the distance and angle between each key point and the second acquisition device are calculated as the second spatial relative position.

7. A process tracking device based on equipment maintenance, characterized in that, The device includes: The personnel data acquisition module is used to acquire personnel-related data of maintenance personnel in the maintenance scenario. The personnel-related data includes the posture category of the maintenance personnel and the first spatial relative position of the maintenance personnel. The first spatial relative position is the spatial relative position between the maintenance personnel and the first acquisition device. The component data acquisition module is used to acquire component-related data in the maintenance scenario. The component-related data includes the component category and the second spatial relative position of the component corresponding to the component category. The second spatial relative position is the relative position of the component and the second acquisition device in the scenario. The behavior information determination module is used to determine the local behavior information of the maintenance personnel towards the component based on the personnel-related data and the component-related data; The process tracking module is used to track the maintenance process of the maintenance scenario based on the local behavior information. The behavior information determination module includes the following sub-modules: The relative position relationship determination submodule is used to determine the relative position relationship between the maintenance personnel and the component based on the first spatial relative position and the second spatial relative position; The local behavior template matching submodule is used to generate matching information based on the posture category, the component category and the relative position relationship, and to match the matching information with a plurality of pre-generated local behavior templates, so as to use the successfully matched local behavior template as the local behavior information of the matching information, wherein each local behavior template includes a template posture category, a template component category and a template relative position relationship; The first spatial relative position includes the distance and angle between the first data acquisition device positioned in the maintenance scenario and the maintenance personnel; the second spatial relative position includes the angle and distance between the second data acquisition device positioned on the maintenance personnel and the component. The relative position relationship determination submodule is specifically used for: The distance and angle between the component and the second acquisition device are taken as the distance and angle between the component and the maintenance personnel. The process tracking module is specifically used for: Obtain information on one or more local behaviors performed by the maintenance personnel in the maintenance scenario, and organize the information on one or more local behaviors into a list of local behaviors in chronological order; The step sequence of the local behavior list is matched with the step sequence of one or more previously generated process templates, and the successfully matched process template is taken as the process category corresponding to the local behavior list. The component data acquisition module is specifically used for: Obtain a second image data sequence containing the component; Extract the image of the second region where the component is located from each of the second image data; Component identification is performed on a continuous second region image sequence to determine the component category; wherein, a pre-generated component category identification network is used to identify the component category of the second region image sequence; The process template includes multiple process steps, and the process tracking module is also used for: Based on the list of local behaviors, determine whether all process steps in the successfully matched process template have been completed; If so, then the current process category is determined to be complete; If not, then the current process category is determined to be incomplete.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the method of any one of claims 1-6.

Images