Projection optimization method and apparatus for human-machine collaborative assembly, terminal, and medium

Abstract

The present application discloses a projection optimization method and apparatus for human-machine collaborative assembly, a terminal, and a medium. The method comprises: acquiring target initial poses, the target initial poses comprising initial poses of a target observer, a target projection object, and a target projector; constructing a target projection model, and on the basis of the initial poses, performing the mapping of a projection scene in the target projection model; acquiring target key points, and acquiring a target mapping relationship on the basis of the target key points, the target mapping relationship being a relationship between the target projection object and a projected image; constructing a projection layout optimization model on the basis of the target mapping relationship, and calculating a target projection position on the basis of the projection layout optimization model; and adjusting the position of the target projector on the basis of the target projection position. The projection optimization method for human-machine collaborative assembly provided by the present application can effectively improve the flexibility of projection layout, and realize adaptive generation of projection layouts in a human-machine collaborative environment.

Description

Projection optimization method and device for human-machine collaborative assembly, terminal and medium TECHNICAL FIELD

[0001] The present application relates to the technical field of high-precision assembly of large-size complex parts, and particularly relates to a projection optimization method and device for human-machine collaborative assembly, a terminal and a medium. BACKGROUND

[0002] Large-size parts such as wallboards and skins are widely used in complex large products in the aviation, aerospace and shipbuilding industries. Since the parts of complex products have high customization degree, higher requirements are also put on the operation level of assembly personnel. Spatial augmented reality technology can project assembly process information directly onto the surface of a part, which has important engineering significance for helping assembly workers to more conveniently perform assembly operation. At present, scholars mainly adopt a fixed projection layout method to study spatial augmented reality technology in an assembly site, which is mainly used for desktop assembly. This method has poor flexibility. When the projection layout is changed, it takes a long time to reconfigure the projection layout, and the position of the projector needs to be manually adjusted, which is very inconvenient.

[0003] Therefore, the prior art still needs to be improved and enhanced. SUMMARY

[0004] In view of the above defects of the prior art, the projection optimization method and device for human-machine collaborative assembly, the terminal and the medium are provided, aiming to solve the problem that the prior art has poor flexibility when performing human-machine collaborative assembly, and the position of the projector needs to be manually adjusted.

[0005] In a first aspect, the present application provides a projection optimization method for human-machine collaborative assembly, comprising:

[0006] Obtaining a target initial pose, the target initial pose comprising initial poses of a target observer, a target projected object and a target projector;

[0007] Constructing a target projection model, and performing mapping of a projection scene in the target projection model based on the initial poses;

[0008] Obtaining target key points, obtaining a target mapping relationship based on the target key points, the target key points comprising corner points and assembly key points of the target projected object, and the target mapping relationship being a relationship between the target projected object and a projected image;

[0009] Constructing a projection layout optimization model based on the target mapping relationship, and calculating a target projection position based on the projection layout optimization model;

[0010] Adjusting the position of the target projector based on the target projection position.

[0011] The projection optimization method for human-machine collaborative assembly, wherein the obtaining of the target initial pose comprises:

[0012] The field coordinate system is constructed, the initial direction is defined, the position parameters and the attitude parameters of the target observer, the target projected object and the target projector are recorded based on the infrared marker point method as the target initial pose, the position parameters represent the position of the object in the field coordinate system, and the attitude parameters represent the offset angle of the object relative to the initial direction.

[0013] The projection optimization method for human-machine collaborative assembly, wherein the constructing of the target projection model comprises:

[0014] The target projector parameters are obtained, the target projector is modeled based on the target projector parameters by using the pinhole model to obtain the target projector model;

[0015] The target observer is modeled based on the target size of the cylinder to obtain the target observer model;

[0016] The key point and key surface information of the projected object are obtained, the projected object is modeled based on the multiple groups of triangular facet formats formed by the key point and key surface information to obtain the target projected object model;

[0017] The target projection model is constructed based on the target projector model, the target observer model and the target projected object model.

[0018] The projection optimization method for human-machine collaborative assembly, wherein the mapping of the projection scene in the target projection model based on the initial pose comprises:

[0019] The virtual coordinate system same as the field coordinate system is constructed;

[0020] The target projector model, the target observer model and the target projected object model are placed into the virtual coordinate system corresponding to the position parameters based on the position parameters;

[0021] The target projector model, the target observer model and the target projected object model are rotated to the position angle corresponding to the attitude parameters based on the attitude parameters and the initial direction.

[0022] The projection optimization method for human-machine collaborative assembly, wherein the obtaining of the target mapping relationship based on the target key point comprises:

[0023] The target intrinsic matrix and the target extrinsic matrix of the target projector are obtained;

[0024] Calculate a mapping relationship between each target key point and a corresponding two-dimensional pixel point on the projected image based on the target intrinsic matrix and the target extrinsic matrix.

[0025] The projection optimization method for human-machine collaborative assembly, wherein the constructing a projection layout optimization model based on the target mapping relationship comprises:

[0026] Performing inverse operation on the position and size of the projected image based on the target key point, the target mapping relationship and the pose of the target projector to obtain a target projected image.

[0027] Obtaining a target constraint, and constructing the projection layout optimization model with the target projected image as a target based on the target constraint.

[0028] The projection optimization method for human-machine collaborative assembly, wherein the performing inverse operation on the position and size of the projected image based on the target key point, the target mapping relationship and the pose of the target projector to obtain a target projected image comprises:

[0029] Obtaining a target pixel point corresponding to the target key point on the projected image based on the target mapping relationship;

[0030] Transforming the projected image based on the target pixel point to obtain the target projected image satisfying the position of the target pixel point.

[0031] A second aspect of the present application provides a projection optimization device for human-machine collaborative assembly, comprising:

[0032] A pose obtaining module is configured to obtain a target initial pose, wherein the target initial pose comprises initial poses of a target observer, a target projected object and a target projector.

[0033] A constructing module is configured to construct a target projection model, and perform mapping of a projection scene in the target projection model based on the initial pose.

[0034] A mapping module is configured to obtain a target key point, and obtain a target mapping relationship based on the target key point, wherein the target key point comprises corner points and assembly key points of the target projected object, and the target mapping relationship is a relationship between the target projected object and a projected image.

[0035] An optimization module is configured to construct a projection layout optimization model based on the target mapping relationship, and calculate a target projection position based on the projection layout optimization model.

[0036] An adjusting module is configured to adjust the position of the target projector based on the target projection position.

[0037] In a third aspect, the present application provides a terminal, comprising: a processor, and a storage medium connected with the processor, the storage medium being adapted to store a plurality of instructions, and the processor being adapted to invoke the instructions in the storage medium to perform the steps of the projection optimization method for human-machine collaborative assembly.

[0038] In a fourth aspect, the present application provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs are executable by one or more processors to implement the steps of the projection optimization method for human-machine collaborative assembly.

[0039] Beneficial effects: Compared with the prior art, the present application provides a projection optimization method, device, terminal and medium for human-machine collaborative assembly. In the projection optimization method for human-machine collaborative assembly provided by the present application, a target initial pose is obtained, the target initial pose including initial poses of a target observer, a target projected object and a target projector, a target projection model is then constructed, mapping of a projection scene is performed in the target projection model based on the initial poses, further, a target key point is obtained, a target mapping relationship is obtained through the target key point, the target key point including corner points and assembly key points of the target projected object, the target mapping relationship being a relationship between the target projected object and a projected image, then, a projection layout optimization model is constructed based on the target mapping relationship, a target projection position is calculated based on the projection layout optimization model, and finally, the position of the target projector is adjusted based on the target projection position. Through the projection optimization method for human-machine collaborative assembly provided by the present application, the projection scene of large-scale human-machine collaborative assembly is optimized, the flexibility and adaptability of the projection layout are effectively improved, and adaptive generation of the projection layout in the human-machine collaborative environment is realized. BRIEF DESCRIPTION OF DRAWINGS

[0040] FIG. 1 is a flowchart of an embodiment of the projection optimization method for human-machine collaborative assembly provided by the present application;

[0041] FIG. 2 is a schematic diagram of a model structure in an embodiment of the projection optimization method for human-machine collaborative assembly provided by the present application;

[0042] FIG. 3 is a schematic diagram of a projection scene in an embodiment of the projection optimization method for human-machine collaborative assembly provided by the present application;

[0043] FIG. 4 is a schematic structural diagram of an embodiment of the projection optimization device for human-machine collaborative assembly provided by the present application;

[0044] FIG. 5 is a schematic structural diagram of an embodiment of the terminal provided by the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and effect of the present application more clear and explicit, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and not to limit the present application.

[0046] It can be understood by those skilled in the art that, unless specifically stated otherwise, the singular forms "a", "an" and "the" as used herein are intended to include plural forms as well. It should be further understood that the word "comprising" as used in the specification of the present application means that the features, integers, steps, operations, elements and / or components described in the specification exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be an intermediate element. In addition, "connected" or "coupled" as used herein can include wireless connection or wireless coupling. The phrase "and / or" as used herein includes all or any one of the associated listed items and all combinations thereof.

[0047] It can be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0048] The projection optimization method for human-machine collaborative assembly provided by the present application can be applied to a terminal with computing capability, which can execute the projection optimization method for human-machine collaborative assembly provided by the present application to detect and position the target position in the image to be processed.

[0049] Embodiment one

[0050] At present, with the development of technologies such as augmented reality, digital twin, IoT, and robots, assembly assistance technology gradually changes from passive information display to active adaptive guidance. Under the concept of human-machine collaboration, digital twin technology can provide a fusion and complementary adaptive augmented reality guidance method for human-robot collaborative space. Therefore, in the present embodiment, a projection optimization method for human-machine collaborative assembly is provided, which performs adaptive projection layout optimization in the projection scene of digital twin, improves the flexibility of the projection system, reduces the time consumed for reconfiguring the projection layout, and better supports human-machine collaborative assembly.

[0051] Specifically, in the embodiment, a projection optimization method for human-robot collaborative assembly is provided. As shown in FIG. 1, the projection optimization method for human-robot collaborative assembly provided by the present application comprises the following steps:

[0052] S100, obtaining a target initial pose, the target initial pose comprising initial poses of a target observer, a target projected object and a target projector.

[0053] Referring to FIG. 2, in the embodiment, the real scene is mapped to a terminal with computing capability based on a projection scene virtual-real mapping module, the projected image is reversely generated in the virtual scene based on a key point reverse image generation module after the real scene is mapped to the virtual scene, and finally, the projection layout is optimized based on a multi-element projection layout optimization module. Specifically, in the projection scene virtual-real mapping stage, first, the positions of the observer, the projected object and the projector in the real projection scene are obtained.

[0054] The obtaining of the target initial pose comprises:

[0055] The positions and attitude parameters of the target observer, the target projected object and the target projector are recorded as the target initial pose based on an infrared marker point method, the position parameter representing the position of the object in the field coordinate system, and the attitude parameter representing the offset angle of the object relative to the initial direction.

[0056] When the projection scene of human-robot collaborative assembly is optimized, the field situation needs to be obtained first, the real scene information is collected, and the environment of the manufacturing field is dynamically captured by an infrared camera to construct the field coordinate system.

[0057] Specifically, in the infrared camera motion capture environment of the manufacturing field, a projection coordinate system is constructed with the center ground position of the manufacturing field as the origin, the vertical ground direction as the Z axis, and the two ground directions perpendicular to each other as the X and Y axes. In the projection coordinate system, the position parameters and the attitude parameters of the marker points of the target observer O, the surface S of the target projected object and the target projector P are obtained by the infrared marker point method. The position parameter represents the position of the object in the field coordinate system, and in the field coordinate system, the distance of the object from the origin in the X, Y and Z directions is represented by (x n , y n , z n ); the attitude parameter represents the offset angle of the object relative to the initial direction, and the angles of rotation of the object relative to the initial direction in the X, Y and Z axes are represented by (α n , β n , γ n ), and specifically, the initial direction is a default direction vector set in advance.

[0058] Specifically, the position parameters of a set of marker points and marker surfaces and the attitude parameters are used to represent the attitude of a rigid object, i.e., the target projected object, in a projection space; for the target observer, the center position in the site plane is determined by the marker points of the torso position without the need to obtain the attitude parameters; and the light center position of the target projector is obtained as the position parameter, and the attitude parameters of the projector need to be obtained.

[0059] S200, constructing a target projection model, and mapping the projection scene in the target projection model based on the initial pose.

[0060] The target projection model is constructed, including:

[0061] Obtaining target projector parameters, modeling the target projector based on the target projector parameters using a pinhole model to obtain a target projector model;

[0062] Modeling the target observer based on a cylinder with a target size to obtain a target observer model;

[0063] Obtaining key point and key surface information of the projected object, modeling the projected object based on a plurality of sets of triangular facet formats formed by the key point and key surface information to obtain a target projected object model;

[0064] Constructing the target projection model based on the target projector model, the target observer model, and the target projected object model.

[0065] Specifically, mapping the real scene to the virtual scene also includes obtaining the target projection parameters, the target projection parameters including a projector projection ratio and size information of the projected image, and constructing a three-dimensional virtual projection space that maps the physical space of the real scene based on the target projection parameters.

[0066] In the embodiment, the modeling of the target projector is the modeling of the projection process using a pinhole model, the position of the optical center of the projector is taken as the center of the pinhole imaging, and the projected image pixel points are projected through the optical center according to the principle of straight-line propagation of light, wherein the projection ratio of the projector determines the projection range, and the size of the projected image determines the size of the projection augmented information; for the target observer, a cylindrical region at the position of the torso center is used to represent the human body shielding of the projection, and when the projection simulation path is shielded by the cylinder corresponding to the observer, the projection is interfered by the observer; for the projected object, the spatial geometric information is expressed in the triangular facet format containing point and surface information of the corresponding three-dimensional model, and when the projected image pixel intersects with the triangular facet of the three-dimensional model, the projection process is completed.

[0067] The mapping of the projection scene in the target projection model based on the initial pose comprises:

[0068] A virtual coordinate system identical to the live coordinate system is constructed;

[0069] The target projector model, the target observer model and the target projected object model are placed in the virtual coordinate system at positions corresponding to the position parameters based on the position parameters;

[0070] The target projector model, the target observer model and the target projected object model are rotated to position angles corresponding to the pose parameters based on the pose parameters and the initial direction.

[0071] That is, the mapping of the real scene to the virtual scene further comprises the mapping of the projection scene. Specifically, a virtual coordinate system identical to the live coordinate system is constructed, and for the target projector, the target observer and the target projected object, the marker point positions in the initial pose are spatially pose-matched with the target initial pose, and the spatial pose is calculated.

[0072] The target projector model, the target observer model and the target projected object model are placed in the virtual coordinate system at positions corresponding to the position parameters based on the position parameters, and the target projector model, the target observer model and the target projected object model are rotated to position angles corresponding to the pose parameters based on the pose parameters and the initial direction.

[0073] Specifically, the initial positions of the elements are (0, 0, 0) by default, the projection direction of the projector is (0, 1, 0), and the pose of the projected object is the same as that in the design coordinate system. For the projector, the observer and the projected object, similarly, the position parameters (x, y, z) are used to determine the positions of the elements in the virtual coordinate system, and the pose parameters (a, b, c) are used to determine the rotation angles of the elements in the virtual coordinate system.n , y n , z n ) represents the distance of the nth point or surface from the origin in the projection space coordinate system on the X, Y, Z axes, and the attitude parameters (a n , b n , g n ) represent the angles of rotation thereof relative to the initial direction on the X, Y, Z axes, and the mapped scene graph is shown in FIG. 3.

[0074] S300, obtain a target key point, and obtain a target mapping relationship based on the target key point, the target key point including a corner point and an assembly key point of the target projected object, and the target mapping relationship being a relationship between the target projected object and a projected image.

[0075] The target mapping relationship based on the target key point includes:

[0076] Obtain a target intrinsic matrix and a target extrinsic matrix of the target projector.

[0077] Calculate a mapping relationship between each target key point and a corresponding two-dimensional pixel point on the projected image based on the target intrinsic matrix and the target extrinsic matrix.

[0078] Specifically, for the target projected object, a spatial projection point cloud on the target projected object is extracted, a series of key points PC T ={P T(1) , P T(2) ,..., P T(n)} are selected therefrom, the key points can be corner points or points required by engineering on the target projected object, and are used to ensure that a real effect in projection meets expectations.

[0079] Then, the target intrinsic matrix and the target extrinsic matrix of the target projector are obtained, and a mapping relationship [u, v, 1] = M Int M Ext [x W , y W , z W , 1] is calculated in a matrix calculation manner from a three-dimensional projection point [x W , y W , z W ] in a projection space coordinate system to a two-dimensional pixel [u, v] on a two-dimensional projected image of the projector. Int M Ext [x W , y W , z W , 1],

[0080] M Ext is the target extrinsic matrix:

[0081] wherein R and T represent rotation and translation parameters of the projector in the projection space, and can be calculated according to a change relationship between the 6 degrees of freedom corresponding to the target initial pose parameters (x M , y M , z M , a M , b M , g M ) of the target projector and the target initial pose parameters corresponding to the target projector.

[0082] M Int is the target intrinsic matrix:

[0083] wherein f represents the focal length of the projector, dx and dy represent the real length of a unit pixel, u0 and v0 represent the number of pixels in the width and height of the image. After the target key point information required by all three-dimensional space projection is mapped to the key pixel of the projected image in two dimensions, a key pixel list L on the projected image in two dimensions can be obtained, which contains the positions of all pixel points in the projected image in a certain order, i.e., the target mapping relationship.

[0084] S400, constructing a projection layout optimization model based on the target mapping relationship, and calculating a target projection position based on the projection layout optimization model.

[0085] The constructing of the projection layout optimization model based on the target mapping relationship comprises:

[0086] S410, performing inverse operation on the position and size of the projected image based on the target key point, the target mapping relationship and the pose of the target projector, to obtain a target projected image.

[0087] The inverse operation on the position and size of the projected image based on the target key point, the target mapping relationship and the pose of the target projector to obtain a target projected image comprises:

[0088] S411, obtaining a target pixel point corresponding to the target key point on the projected image based on the target mapping relationship.

[0089] S412, transforming the projected image based on the target pixel point to obtain the target projected image satisfying the position of the target pixel point.

[0090] Specifically, in the projected image, the original key pixel point position is represented by an initial pixel list L0, and the order is the same as that in the key pixel list L. According to the correspondence between the key pixel list L and the position in the original key pixel list L0, the transformation of the projected image is performed. For the projected image generation on the curved surface part, a thin plate spline interpolation algorithm can be adopted, and for the projected image generation on the plane, an affine transformation interpolation algorithm can be adopted, so as to obtain the inverse operation of the space augmented reality result to the target projected image in two dimensions.

[0091] S420, obtaining a target constraint, and constructing the projection layout optimization model based on the target projected image as a target.

[0092] In the embodiment, the projection layout optimization model is:

[0093] Wherein, the optimization target is to maximize the area ratio of the projected image; constraint 1 is the focal length range of the projector; constraint 2 is the range of the position of the projector not being blocked by people; constraint 3 is the integrity of the projection result; and constraint 4 is that the position of the projector is within the target reachable projection domain.

[0094] Specifically, the area ratio of the target projected image is maximized as a target, and the best projection position is found in the target reachable projection domain, which includes four constraints. Constraint 1 represents the focal length position of the projector. Constraint 2 represents that the projector is not blocked by people. Constraint 3 represents that the projection result has integrity. Constraint 4 represents that the position of the projector is within the reachable area. The projection layout optimization model is:

[0095] Wherein, the optimization target is to maximize the area ratio of the projected image; constraint 1 is the focal length range of the projector; constraint 2 is the range of the position of the projector not being blocked by people; constraint 3 is the integrity of the projection result; and constraint 4 is that the position of the projector is within the target reachable projection domain.

[0096] Based on the projection layout optimization model, the area ratio of the target projected image is calculated under the four projection layout constraints. The position corresponding to the largest target projected image result is reserved as the projection layout optimization result, so as to output the target projection position.

[0097] S500, adjusting the position of the target projector based on the target projection position.

[0098] Specifically, the target projection position is the guide position of the target projector in the current projection scene, and the target projector can be adjusted to the target projection position by automatic or manual means.

[0099] It can be seen that the embodiment projects the assembly scene of a large component in reality one-to-one into a virtual scene through a series of operations, and then in the virtual scene, based on the three-dimensional projected object, the size and shape of the corresponding two-dimensional projected image of the projector in each position within the reachable projection domain are inferred reversely, an optimization model is constructed with the maximum area ratio of the projected image as the target, the best projection position is obtained, and the projector is moved to the best projection position to complete the projection optimization for human-machine collaborative assembly.

[0100] In this way, by constructing the projection simulation scene of virtual-real mapping, the optical characteristics of the projection process are depicted in the virtual scene, providing a basis for the evaluation of the projection layout. Then, by using the mapping method of three-dimensional projection feature points and two-dimensional projected image feature pixels, the reverse generation of the projected image is realized. Further, by constructing the projection layout optimization model, the adaptive generation of the projection layout in the human-machine collaborative environment is realized, solving the problems of poor flexibility and low adaptability of the projection layout in the spatial augmentation assistance process of large-scale products.

[0101] To sum up, the embodiment provides a projection optimization method for human-machine collaborative assembly, which acquires a target initial pose, the target initial pose including initial poses of a target observer, a target projected object and a target projector, then constructs a target projection model, maps a projection scene in the target projection model based on the initial pose, further acquires target key points, acquires a target mapping relationship through the target key points, the target key points including corner points and assembly key points of the target projected object, the target mapping relationship being a relationship between the target projected object and a projected image, then constructs a projection layout optimization model based on the target mapping relationship, calculates a target projection position based on the projection layout optimization model, and finally adjusts the position of the target projector based on the target projection position. The projection optimization method for human-machine collaborative assembly provided by the embodiment optimizes the projection scene of large-scale human-machine collaborative assembly, effectively improves the flexibility and adaptability of the projection layout, and realizes the adaptive generation of the projection layout in the human-machine collaborative environment.

[0102] It should be understood that although each step in the flowchart shown in the drawings of the present application specification is shown in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless explicitly stated herein, the execution of the steps in the present application has no strict order limitation, and these steps can be executed in other orders. Moreover, at least part of the steps of the present application can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or sub-steps or stages of other steps.

[0103] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0104] Embodiment two

[0105] Based on the above-mentioned embodiments, the present application further provides a projection optimization device for human-machine collaborative assembly, and a functional module schematic diagram is shown in FIG. 4. The projection optimization device for human-machine collaborative assembly includes:

[0106] A pose acquisition module is configured to acquire a target initial pose, wherein the target initial pose includes initial poses of a target observer, a target projected object and a target projector, and the details are described in Embodiment One.

[0107] a construction module, configured to construct a target projection model, and perform mapping of the projection scene in the target projection model based on the initial pose;

[0108] a mapping module, configured to obtain target key points, and obtain a target mapping relationship based on the target key points, the target key points including corner points and assembly key points of the target projected object, and the target mapping relationship being a relationship between the target projected object and the projected image;

[0109] an optimization module, configured to construct a projection layout optimization model based on the target mapping relationship, and calculate a target projection position based on the projection layout optimization model;

[0110] an adjustment module, configured to adjust the position of the target projector based on the target projection position.

[0111] Embodiment Three

[0112] Based on the projection optimization method for human-machine collaborative assembly described in Embodiment One, the application further provides a terminal, and a principle block diagram of the terminal can be as shown in FIG. 5. The terminal includes a memory 10 and a processor 20, and the memory 10 stores a projection optimization program for human-machine collaborative assembly. When the processor 10 executes the computer program, at least the following steps can be implemented:

[0113] obtain a target initial pose, the target initial pose including initial poses of a target observer, a target projected object and a target projector;

[0114] construct a target projection model, and perform mapping of the projection scene in the target projection model based on the initial pose;

[0115] obtain target key points, and obtain a target mapping relationship based on the target key points, the target key points including corner points and assembly key points of the target projected object, and the target mapping relationship being a relationship between the target projected object and the projected image;

[0116] construct a projection layout optimization model based on the target mapping relationship, and calculate a target projection position based on the projection layout optimization model;

[0117] adjust the position of the target projector based on the target projection position.

[0118] The obtaining of the target initial pose includes:

[0119] Construct a scene coordinate system, define an initial direction, record position parameters and attitude parameters of the target observer, the target projected object and the target projector as the target initial pose based on an infrared marker point method, the position parameters representing a position of the object in the scene coordinate system, and the attitude parameters representing an offset angle of the object relative to the initial direction.

[0120] The target projection model is constructed by:

[0121] Obtain target projector parameters, model the target projector based on the target projector parameters using a pinhole model to obtain a target projector model;

[0122] Model the target observer based on a target size cylinder to obtain a target observer model;

[0123] Obtain key point and key surface information of the projected object, model the projected object based on multiple groups of triangular facet formats formed by the key point and key surface information to obtain a target projected object model;

[0124] Construct the target projection model based on the target projector model, the target observer model and the target projected object model.

[0125] The mapping of the projection scene in the target projection model based on the initial pose includes:

[0126] Construct a virtual coordinate system identical to the scene coordinate system;

[0127] Place the target projector model, the target observer model and the target projected object model into the virtual coordinate system at positions corresponding to the position parameters based on the position parameters;

[0128] Rotate the target projector model, the target observer model and the target projected object model to position angles corresponding to the attitude parameters based on the attitude parameters and the initial direction.

[0129] The target mapping relationship is obtained based on the target key points, including:

[0130] Obtain a target intrinsic matrix and a target extrinsic matrix of the target projector;

[0131] Calculate a mapping relationship between each target key point and a corresponding two-dimensional pixel point on the projected image based on the target intrinsic matrix and the target extrinsic matrix.

[0132] The projection layout optimization model is constructed based on the target mapping relationship, including:

[0133] perform inverse operation on the position and size of the projected image based on the target key point, the target mapping relationship and the pose of the target projector, to obtain a target projected image;

[0134] obtain a target constraint, and construct the projection layout optimization model based on the target projected image being maximized as a target based on the target constraint.

[0135] The inverse operation on the position and size of the projected image based on the target key point, the target mapping relationship and the pose of the target projector, to obtain a target projected image, comprises:

[0136] obtain a target pixel point corresponding to the target key point on the projected image based on the target mapping relationship;

[0137] transform the projected image based on the target pixel point, to obtain the target projected image satisfying the position of the target pixel point.

[0138] Embodiment Four

[0139] The application also provides a storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the projection optimization method for human-machine collaborative assembly described in the above embodiments.

[0140] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A projection optimization method for human-robot collaborative assembly, characterized in that, The method comprises the following steps: acquiring a target initial pose, the target initial pose comprising initial poses of a target observer, a target projected object and a target projector; constructing a target projection model, mapping a projection scene in the target projection model based on the initial poses; acquiring target key points, acquiring a target mapping relationship based on the target key points, the target key points comprising corner points and assembly key points of the target projected object, and the target mapping relationship being a relationship between the target projected object and a projected image; constructing a projection layout optimization model based on the target mapping relationship, and calculating a target projection position based on the projection layout optimization model; adjusting a position of the target projector based on the target projection position.

2. The projection optimization method for human-robot collaboration assembly according to claim 1, characterized in that, The acquiring of the target initial pose comprises the following steps: constructing a field coordinate system, defining an initial direction, recording position parameters and attitude parameters of the target observer, the target projected object and the target projector as the target initial pose based on an infrared marker point method, the position parameters representing positions of objects in the field coordinate system, and the attitude parameters representing offset angles of objects relative to the initial direction.

3. The projection optimization method for human-robot collaboration assembly according to claim 2, characterized in that, The constructing of the target projection model comprises the following steps: acquiring target projector parameters, modeling the target projector based on the target projector parameters to obtain a target projector model by using a pinhole model; modeling the target observer based on a target size cylinder to obtain a target observer model; acquiring key points and key surface information of the projected object, modeling the projected object based on a plurality of groups of triangular facet formats formed by the key points and the key surface information to obtain a target projected object model; constructing the target projection model based on the target projector model, the target observer model and the target projected object model.

4. The projection optimization method for human-robot collaboration assembly according to claim 3, characterized in that, The mapping of the projection scene in the target projection model based on the initial poses comprises the following steps: constructing a virtual coordinate system identical to the field coordinate system; placing the target projector model, the target observer model and the target projected object model into positions corresponding to the position parameters in the virtual coordinate system based on the position parameters; rotating the target projector model, the target observer model and the target projected object model to position angles corresponding to the attitude parameters based on the attitude parameters and the initial direction.

5. The projection optimization method for human-robot collaboration assembly according to claim 1, wherein, The acquiring of the target mapping relationship based on the target key points comprises the following steps: acquiring a target intrinsic matrix and a target extrinsic matrix of the target projector; calculating a mapping relationship between each target key point and a corresponding two-dimensional pixel point on the projected image based on the target intrinsic matrix and the target extrinsic matrix.

6. The method of claim 1, wherein, The constructing of the projection layout optimization model based on the target mapping relationship comprises the following steps: performing inverse operations on positions and sizes of the projected image based on the target key points, the target mapping relationship and a pose of the target projector to obtain a target projected image; acquiring target constraints, and constructing the projection layout optimization model with the target projected image being maximized as a target based on the target constraints.

7. The projection optimization method for human-robot collaboration assembly according to claim 6, characterized in that, The inverse operation of the position and size of the projected image based on the target key point, the target mapping relationship and the pose of the target projector obtains a target projected image, and the inverse operation includes: Obtaining a target pixel point corresponding to the target key point on the projected image based on the target mapping relationship; Transforming the projected image based on the target pixel point to obtain the target projected image satisfying the position of the target pixel point.

8. A projection optimization device for human-robot collaboration assembly, characterized by, The device includes: A pose acquisition module configured to acquire a target initial pose, the target initial pose including initial poses of a target observer, a target projected object and a target projector; A construction module configured to construct a target projection model and perform mapping of a projection scene in the target projection model based on the initial pose; A mapping module configured to acquire a target key point, acquire a target mapping relationship based on the target key point, the target key point including corner points and assembly key points of the target projected object, and the target mapping relationship being a relationship between the target projected object and a projected image; An optimization module configured to construct a projection layout optimization model based on the target mapping relationship and calculate a target projection position based on the projection layout optimization model; An adjustment module configured to adjust the position of the target projector based on the target projection position.

9. A terminal, characterized by comprising: The terminal includes a processor, a storage medium in communication connection with the processor, the storage medium being adapted to store a plurality of instructions, and the processor being adapted to invoke the instructions in the storage medium to perform steps of the projection optimization method for human-machine collaborative assembly according to any one of claims 1-7.

10. A storage medium, characterized by The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the projection optimization method for human-machine collaborative assembly according to any one of claims 1-7.

Images