Arrowhead optimization identification method and device

Abstract

The invention relates to an arrow optimization identification method and device, and the method comprises the steps: identifying and storing a first arrow, including preprocessing, initial feature line processing, and first arrow reconstruction; and obtaining a comparison object based on the reconstructed first arrow, extracting a 2D feature line of a second arrow, matching the 2D feature line with the comparison object, and recovering the pose of the second arrow by adopting an EPnL algorithm according to a matching result. Compared with the prior art, the method has the advantages of high universality, simple identification process and the like.

Description

Technical Field

[0001] The present invention relates to the field of visual processing, and in particular to a method and device for optimizing arrow recognition. Background Art

[0002] There are many arrow structures in the underground garage scene, such as straight arrows, left turn arrows, etc. All arrows play an important role in whether the driver can accurately find a parking space or enter and exit the garage, so it is necessary to accurately identify various arrows. In the existing technology of arrow recognition, it mainly relies on deep learning for detection and reconstruction. Such methods require a large number of data set samples and are not very general. Alternatively, the RANSAC method is used to perform plane fitting on the feature points belonging to the arrow, but due to the low texture characteristics of the arrow itself, the point features are also less distributed, which affects the accuracy of the plane fitting and even affects the performance of the system during back-end optimization.

[0003] Chinese patent application "CN116503839A" discloses a method for identifying road arrows. According to the above assumption, the method first uses a preset arrow model to train the model. In the subsequent arrow recognition process, the regional scene image is input into the model to realize arrow recognition in the scene. Through the trained learning model, accurate recognition of various types of arrows is achieved and the recognition process of the same arrow in the subsequent process is simplified. However, in real life, the specifications of arrows in each underground garage are different. If this method is used, the model needs to be retrained to make it conform to the current scene.

[0004] Therefore, simplifying the arrow recognition process while improving the versatility of the method is a problem that needs to be solved. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method and device for optimizing arrow recognition, which utilizes the edge information of the arrow, models and recognizes the arrow through straight line features, and estimates its position and posture, thereby incorporating the arrow as an object into map management.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, there is provided an arrow optimization recognition method, the method comprising:

[0008] S1: Identify and save the first arrow. The first arrow refers to all arrows of different specifications that appear for the first time during the identification process. The specific steps are as follows:

[0009] Preprocessing: Segment the collected scene image to obtain each object in the scene image, and set the object name and rectangular bounding box attributes; filter objects whose name attributes are arrows as objects to be processed, extract the initial feature lines in the rectangular bounding boxes of the objects to be processed, and assign semantic information to the feature lines; the semantic information is the arrow ID;

[0010] Initial feature line processing: multiple feature lines with the same arrow ID are selected based on semantic information, and the multiple feature lines are converted into 3D to obtain multiple 3D feature lines; point sampling is performed on the multiple 3D feature lines to obtain a sample point set in three-dimensional space;

[0011] Reconstruction of the first arrow: Based on the sample point set in the three-dimensional space, the main direction and the secondary direction of the arrow are calculated based on the PCA algorithm; a new coordinate system is established in the world coordinate system based on the main and secondary directions, and the arrow position is restored in the coordinate system to achieve reconstruction of the first arrow and save the reconstruction result;

[0012] S2: Simplify the identification of the second arrow based on the first arrow, where the second arrow refers to an arrow with the same specifications as the first arrow that appears repeatedly during the identification process.

[0013] The specific steps are as follows: obtain a comparison object based on the reconstructed first arrow, extract the 2D feature line of the second arrow and match it with the comparison object, and restore the position and posture of the second arrow using the EPnL algorithm based on the matching result.

[0014] As a preferred technical solution, the SAM model is used to segment the collected images.

[0015] As a preferred technical solution, a method for realizing 3D conversion of initial feature lines is: adding arrows and semantic information as binary edges of multiple feature lines corresponding to arrow IDs to obtain multiple 3D feature lines.

[0016] As a preferred technical solution, the number of the characteristic lines is at least three.

[0017] As a preferred technical solution, the first arrow reconstruction is specifically as follows:

[0018] Construct a sample matrix according to the sample point set to calculate the column mean, obtain a standardized matrix based on the column mean, process the standardized matrix to obtain a covariance matrix, obtain the eigenvectors and eigenvalues ​​of the covariance matrix, and sort the eigenvalues;

[0019] The eigenvector direction corresponding to the largest eigenvalue is selected as the main direction, and the eigenvector direction corresponding to the second largest eigenvalue is selected as the secondary direction;

[0020] The main and secondary directions are normalized to obtain the main vector and the secondary vector. The main vector is set as the x-axis, the secondary vector is set as the y-axis, the result vector of the main vector cross-multiplied by the secondary vector is used as the z-axis, the mean of the sample points is calculated, and this value is used as the coordinate origin.

[0021] As a preferred technical solution, the formula for calculating the column mean is:

[0022]

[0023] in represents the column mean of the sample point set, n represents the number of columns in the sample matrix, and p ij Represents the element in the i-th row and j-th column of the sample matrix;

[0024] As a preferred technical solution, the formula for calculating each element of the normalized matrix is:

[0025]

[0026] Where S ij represents the element in the i-th row and j-th column of the normalized matrix, p ij represents the element in the i-th row and j-th column of the sample matrix, represents the column mean of the sample point set, and n represents the number of columns of the sample matrix;

[0027] As a preferred technical solution, the formula for calculating each element of the covariance matrix is:

[0028]

[0029] Among them, r ij is the element in the i-th row and j-th column of the covariance matrix, n represents the number of columns in the standardized matrix, S ki represents the element in the kth row and ith column of the normalized matrix, S kj Represents the element at the kth row and jth column in the normalized matrix.

[0030] As a preferred technical solution, S2 is specifically:

[0031] Convert the Plücker coordinates of the first arrow in the world coordinate system to the Plücker coordinates in the arrow coordinate system, and extract the 3D feature line of the first arrow in the arrow coordinate system as a comparison object;

[0032] Extracting the 2D feature line of the second arrow in the current scene image and performing 3D-2D matching with the comparison object;

[0033] The current position and posture of the second arrow is calculated based on the EPnL algorithm, and the three-dimensional model of the second arrow is restored.

[0034] According to a second aspect of the present invention, there is provided a device for optimizing arrow recognition, the device comprising:

[0035] The first module is used for preprocessing the first arrow, which refers to all arrows of different specifications that appear for the first time in the recognition process. The collected scene image is segmented to obtain each object in the scene image, and the object name and rectangular bounding box attributes are set; objects with the name attribute of arrow are selected as objects to be processed, and the initial feature lines in the rectangular bounding box of the objects to be processed are extracted, and semantic information is assigned to the feature lines; the semantic information is the arrow ID;

[0036] The second module is used for initial feature line processing of the first arrow, screening out multiple feature lines with the same arrow ID according to semantic information, converting the multiple feature lines into 3D to obtain multiple 3D feature lines; performing point sampling on the multiple 3D feature lines to obtain a sample point set in three-dimensional space;

[0037] The third module is used for reconstructing the first arrow, and according to the sample point set in the three-dimensional space, the main direction and the secondary direction of the arrow are calculated based on the PCA algorithm; a new coordinate system is established in the world coordinate system according to the main and secondary directions, and the arrow posture is restored in the coordinate system to realize the reconstruction of the first arrow and save the reconstruction result;

[0038] The fourth module is used for second arrow recognition. The second arrow refers to the arrow with the same specifications as the first arrow that appears repeatedly during the recognition process. The second arrow recognition is simplified based on the first arrow. The specific steps are as follows: obtain a comparison object based on the reconstructed first arrow, extract the 2D feature line of the second arrow and match it with the comparison object, and use the EPnL algorithm to restore the second arrow's position and posture based on the matching result.

[0039] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0040] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described when executed by a processor.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] 1) The first arrow collected in the scene is used as a template. When arrows of the same specifications are subsequently identified, there is no need to rebuild them. Instead, they only need to be compared with the template arrows, which simplifies the arrow recognition process and improves the arrow recognition rate.

[0043] 2) This method does not rely on learning models to achieve arrow detection and reconstruction. It is suitable for different underground garage scenarios and has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flow chart of a method for optimizing arrow recognition according to the present invention;

[0045] Figure 2 It is a schematic diagram of the segmentation result of the present invention;

[0046] Figure 3 This is a schematic diagram of 3D feature line sampling of the present invention;

[0047] Figure 4 This is a schematic diagram of the main and secondary directions of arrows in the present invention;

[0048] Figure 5 The first arrow PCA plane fitting and optimization schematic diagram of the present invention;

[0049] Figure 6 This is a reconstruction effect diagram of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0051] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0052] Embodiment 1:

[0053] This embodiment provides a method for optimizing arrow recognition, and its process is as follows: Figure 1 As shown, the specific steps are:

[0054] S1: Identify and save the first arrow, where the first arrow refers to all arrows of different specifications that appear for the first time during the identification process. The specific steps are as follows:

[0055] S11, preprocessing: Use the SAM model to segment the collected image, obtain each object in the scene image, and set the object name and rectangular bounding box attributes; filter the objects with the name attribute of arrow as the objects to be processed, extract the initial feature lines in the rectangular bounding box of the objects to be processed, and assign semantic information to the feature lines; the semantic information is the arrow ID; the effect after segmentation is as follows Figure 2 As shown;

[0056] S12, initial feature line processing: three feature lines with the same arrow ID are selected based on semantic information, and two-dimensional edges of the three feature lines with arrows and semantic information corresponding to the arrow ID are added to obtain three 3D feature lines; point sampling is performed on the three 3D feature lines to obtain a sample point set in three-dimensional space. The collection situation is as follows Figure 3 As shown, the red solid line part in the figure is sampled;

[0057] S13. The first arrow reconstruction specifically includes:

[0058] S131, constructing a sample matrix according to the sample point set to calculate the column mean, obtaining a standardized matrix according to the column mean, processing the standardized matrix to obtain a covariance matrix, obtaining the eigenvectors and eigenvalues ​​of the covariance matrix, and sorting the eigenvalues;

[0059] The formula to calculate the column mean is:

[0060]

[0061] in represents the column mean of the sample point set, n represents the number of columns in the sample matrix, and p ij Represents the element in the i-th row and j-th column of the sample matrix;

[0062] The formula for calculating each element of the normalized matrix is:

[0063]

[0064] Where S ij represents the element in the i-th row and j-th column of the normalized matrix, p ij represents the element in the i-th row and j-th column of the sample matrix, represents the column mean of the sample point set, and n represents the number of columns of the sample matrix;

[0065] The formula for calculating each element of the covariance matrix is:

[0066]

[0067] Among them, r ij is the element in the i-th row and j-th column of the covariance matrix, n represents the number of columns in the standardized matrix, S ki represents the element in the kth row and ith column of the normalized matrix, S kj Represents the element at the kth row and jth column in the normalized matrix.

[0068] S132, selecting the eigenvector direction corresponding to the largest eigenvalue as the main direction, and the eigenvector direction corresponding to the second largest eigenvalue as the secondary direction;

[0069] S133, normalize the main and secondary directions to obtain a main vector and a secondary vector, the main vector is set as the x-axis, the secondary vector is set as the y-axis, the result vector of the main vector cross-multiplied by the secondary vector is used as the z-axis, calculate the mean of the sample points, and use the value as the origin of the coordinates. Figure 4 As shown, the direction indicated by the red arrow is the main direction, and the direction indicated by the green arrow is the secondary direction.

[0070] After step S133, a schematic diagram of the first arrow after plane fitting and optimization based on the PCA algorithm can be obtained, as shown in FIG. Figure 5 shown.

[0071] S2: Simplify the identification of the second arrow based on the first arrow, where the second arrow refers to an arrow with the same specifications as the first arrow that appears repeatedly during the identification process, specifically including:

[0072] S21, converting the Plücker coordinates of the first arrow in the world coordinate system into the Plücker coordinates in the arrow coordinate system, and extracting the 3D feature line of the first arrow in the arrow coordinate system as a comparison object;

[0073] S22, extracting the 2D feature line of the second arrow in the current scene image and performing 3D-2D matching with the comparison object;

[0074] S23. Calculate the current position and posture of the second arrow based on the EPnL algorithm, and restore the three-dimensional model of the second arrow.

[0075] The process of calculating the pose of the second arrow based on the EPnL algorithm in step S23 refers to the relevant process in the article "EPnL: An Efficient and Accurate PnL Problem Solving Algorithm", which will not be repeated in this embodiment.

[0076] The above is an introduction to a method embodiment. The following is a further explanation of the solution of the present invention through an apparatus embodiment.

[0077] Embodiment 2:

[0078] This embodiment provides a device for arrow optimization recognition, which is used to implement the above method. The device is placed on the car. When entering the underground garage, the device starts to work. The working process of each module of the device refers to the method flow in Example 1. Finally, the display terminal displays the following: Figure 6 The effect shown is that the arrow is red, the green is the road, and the purple is other things in the scene.

[0079] The present embodiment also provides an electronic device, including a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or computer program instructions loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0080] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0081] The processing unit performs the various methods and processes described above, such as methods S1 to S2. For example, in some embodiments, methods S1 to S12 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S2 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S2 in any other appropriate manner (e.g., by means of firmware).

[0082] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0083] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0084] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0085] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0086] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An arrow optimization recognition method, It is characterized in that The method includes: S1: Identify and save the first arrow. The first arrow refers to all arrows of different specifications that appear for the first time during the identification process. The specific steps are as follows: Preprocessing: Segment the collected scene image to obtain each object in the scene image, and set the object name and rectangular bounding box attributes; filter objects whose name attributes are arrows as objects to be processed, extract the initial feature lines in the rectangular bounding boxes of the objects to be processed, and assign semantic information to the feature lines; the semantic information is the arrow ID; Initial feature line processing: multiple feature lines with the same arrow ID are selected based on semantic information, and the multiple feature lines are converted into 3D to obtain multiple 3D feature lines; point sampling is performed on the multiple 3D feature lines to obtain a sample point set in three-dimensional space; Reconstruction of the first arrow: Based on the sample point set in the three-dimensional space, the main direction and the secondary direction of the arrow are calculated based on the PCA algorithm; a new coordinate system is established in the world coordinate system based on the main and secondary directions, and the arrow position is restored in the coordinate system to achieve reconstruction of the first arrow and save the reconstruction result; S2: Simplify the identification of the second arrow based on the first arrow, where the second arrow refers to an arrow with the same specifications as the first arrow that appears repeatedly during the identification process. The specific steps are as follows: obtain a comparison object based on the reconstructed first arrow, extract the 2D feature line of the second arrow and match it with the comparison object, and restore the position and posture of the second arrow using the EPnL algorithm based on the matching result.

2. The arrow optimization recognition method according to claim 1, It is characterized in that In the preprocessing step, the SAM model is used to segment the collected image.

3. The arrow optimization recognition method according to claim 1, It is characterized in that The method for realizing 3D conversion of the initial feature lines is: adding arrows and semantic information as binary edges of multiple feature lines corresponding to the arrow IDs to obtain multiple 3D feature lines.

4. The arrow optimization recognition method according to claim 3, It is characterized in that The number of the characteristic lines is at least three.

5. The arrow optimization recognition method according to claim 1, It is characterized in that The first arrow reconstruction is specifically as follows: Construct a sample matrix according to the sample point set to calculate the column mean, obtain a standardized matrix based on the column mean, process the standardized matrix to obtain a covariance matrix, obtain the eigenvectors and eigenvalues ​​of the covariance matrix, and sort the eigenvalues; The eigenvector direction corresponding to the largest eigenvalue is selected as the main direction, and the eigenvector direction corresponding to the second largest eigenvalue is selected as the secondary direction; The main and secondary directions are normalized to obtain the main vector and the secondary vector. The main vector is set as the x-axis, the secondary vector is set as the y-axis, the result vector of the main vector cross-multiplied by the secondary vector is used as the z-axis, the mean of the sample points is calculated, and this value is used as the coordinate origin.

6. The arrow optimization recognition method according to claim 5, It is characterized in that The formula to calculate the column mean is: in represents the column mean of the sample point set, n represents the number of columns in the sample matrix, and p ij Represents the element in the i-th row and j-th column of the sample matrix; The formula for calculating each element of the normalized matrix is: Where S ij represents the element in the i-th row and j-th column of the normalized matrix, p ij represents the element in the i-th row and j-th column of the sample matrix, represents the column mean of the sample point set, and n represents the number of columns of the sample matrix; The formula for calculating each element of the covariance matrix is: Among them, r ij is the element in the i-th row and j-th column of the covariance matrix, n represents the number of columns in the standardized matrix, S ki represents the element in the kth row and ith column of the normalized matrix, S kj Represents the element at the kth row and jth column in the normalized matrix.

7. The arrow optimization recognition method according to claim 1, It is characterized in that The S2 is specifically: Convert the Plücker coordinates of the first arrow in the world coordinate system to the Plücker coordinates in the arrow coordinate system, and extract the 3D feature line of the first arrow in the arrow coordinate system as a comparison object; Extracting the 2D feature line of the second arrow in the current scene image and performing 3D-2D matching with the comparison object; The current position and posture of the second arrow is calculated based on the EPnL algorithm, and the three-dimensional model of the second arrow is restored.

8. An arrow optimization recognition device, It is characterized in that The device includes: The first module is used for preprocessing the first arrow, which refers to all arrows of different specifications that appear for the first time in the recognition process. The collected scene image is segmented to obtain each object in the scene image, and the object name and rectangular bounding box attributes are set; objects with the name attribute of arrow are selected as objects to be processed, and the initial feature lines in the rectangular bounding box of the objects to be processed are extracted, and semantic information is assigned to the feature lines; the semantic information is the arrow ID; The second module is used for initial feature line processing of the first arrow, screening out multiple feature lines with the same arrow ID according to semantic information, converting the multiple feature lines into 3D to obtain multiple 3D feature lines; performing point sampling on the multiple 3D feature lines to obtain a sample point set in three-dimensional space; The third module is used for reconstructing the first arrow, and according to the sample point set in the three-dimensional space, the main direction and the secondary direction of the arrow are calculated based on the PCA algorithm; a new coordinate system is established in the world coordinate system according to the main and secondary directions, and the arrow posture is restored in the coordinate system to realize the reconstruction of the first arrow and save the reconstruction result; The fourth module is used for second arrow recognition. The second arrow refers to the arrow with the same specifications as the first arrow that appears repeatedly during the recognition process. The second arrow recognition is simplified based on the first arrow. The specific steps are as follows: obtain a comparison object based on the reconstructed first arrow, extract the 2D feature line of the second arrow and match it with the comparison object, and use the EPnL algorithm to restore the second arrow's position and posture based on the matching result.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory. It is characterized in that When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images