Method and device for optimizing arrow recognition

Abstract

The present invention relates to a method and device for optimized arrow recognition. The method comprises: identifying and saving a first arrow, including preprocessing, initial feature line processing, and reconstruction of the first arrow; obtaining a comparison object based on the reconstructed first arrow, extracting the 2D feature lines of a second arrow and matching them with the comparison object; and restoring the second arrow's position using an EPnL algorithm based on the matching results. Compared with existing technologies, the present invention has the advantages of high versatility and a simple recognition process.

Description

Technical Field

[0001] The present invention relates to the field of visual processing, and in particular to a method and device for optimizing arrow recognition. Background Art

[0002] Underground garage scenes feature a variety of arrow structures, such as straight arrows and left-turn arrows. All of these arrows play a crucial role in determining whether a driver can accurately find a parking space or enter or exit the garage, necessitating accurate recognition of each type of arrow. Existing techniques for arrow recognition primarily rely on deep learning for detection and reconstruction, but such methods require a large dataset of samples and are not universally applicable. Alternatively, the RANSAC method can be used to perform plane fitting on the feature points belonging to the arrows. However, due to the low texture of the arrows themselves, the point features are also sparsely distributed, impacting the accuracy of the plane fitting and even the performance of the system during back-end optimization.

[0003] Chinese patent application "CN116503839A" discloses a method for identifying road arrows. According to the above-mentioned assumption, this method first uses a preset arrow model to train the model. In the subsequent arrow recognition process, the regional scene image is input into the model to realize arrow recognition in the scene. The trained learning model can realize accurate recognition of various types of arrows and simplify the recognition process of the same arrows in the subsequent process. However, in real life, the specifications of the arrows in each underground garage are different. If this method is used, the model needs to be retrained to make it conform to the current scene.

[0004] Therefore, simplifying the arrow recognition process while improving the versatility of the method is a problem that needs to be solved. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method and device for optimizing the recognition of arrows. The arrows are modeled and recognized by straight line features using their edge information and their position and posture are estimated, and the arrows are then included in map management as objects.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, there is provided an arrow optimization recognition method, the method comprising:

[0008] S1: Identify and save the first arrow. The first arrow refers to all arrows of different sizes that appear for the first time during the identification process. The specific steps are as follows:

[0009] Preprocessing: Segment the captured scene image to obtain each object in the scene image, and set the object name and rectangular bounding box attributes; filter objects with arrow name attributes as objects to be processed, extract the initial feature lines in the rectangular bounding box of the object to be processed, and assign semantic information to the feature lines; the semantic information is the arrow ID;

[0010] Initial feature line processing: multiple feature lines with the same arrow ID are screened out based on semantic information, and the multiple feature lines are converted into 3D to obtain multiple 3D feature lines; point sampling is performed on the multiple 3D feature lines to obtain a sample point set in three-dimensional space;

[0011] First arrow reconstruction: Based on the sample point set in the three-dimensional space, the main direction and the secondary direction of the arrow are calculated using the PCA algorithm; a new coordinate system is established in the world coordinate system based on the main and secondary directions, and the arrow pose is restored in this coordinate system to achieve reconstruction of the first arrow and save the reconstruction result;

[0012] S2: Simplify the identification of the second arrow based on the first arrow. The second arrow refers to the arrow with the same specifications as the first arrow that appears repeatedly during the identification process.

[0013] The specific steps are as follows: a comparison object is obtained based on the reconstructed first arrow, the 2D feature line of the second arrow is extracted and matched with the comparison object, and the EPnL algorithm is used to restore the position of the second arrow according to the matching result.

[0014] As a preferred technical solution, the SAM model is used to segment the collected images.

[0015] As a preferred technical solution, a method for converting the initial feature lines into 3D is to add arrows and semantic information as binary edges of multiple feature lines corresponding to arrow IDs to obtain multiple 3D feature lines.

[0016] As a preferred technical solution, the number of the characteristic lines is at least three.

[0017] As a preferred technical solution, the first arrow reconstruction is specifically as follows:

[0018] Constructing a sample matrix according to the sample point set and calculating the column mean, obtaining a normalized matrix based on the column mean, processing the normalized matrix to obtain a covariance matrix, obtaining the eigenvectors and eigenvalues ​​of the covariance matrix, and sorting the eigenvalues;

[0019] The eigenvector direction corresponding to the largest eigenvalue is selected as the main direction, and the eigenvector direction corresponding to the second largest eigenvalue is selected as the secondary direction;

[0020] Normalize the main and secondary directions to obtain the main vector and the secondary vector. The main vector is set as the x-axis, the secondary vector is set as the y-axis, and the result vector of the main vector cross-multiplied by the secondary vector is used as the z-axis. Calculate the mean of the sample points and use this value as the coordinate origin.

[0021] As a preferred technical solution, the formula for calculating the column mean is:

[0022]

[0023] in represents the column mean of the sample point set, n represents the number of columns of the sample matrix, p ij Represents the element in row i and column j of the sample matrix;

[0024] As a preferred technical solution, the formula for calculating each element of the normalized matrix is:

[0025]

[0026] Among them S ij represents the element in row i and column j of the normalized matrix, p ij represents the element in row i and column j of the sample matrix, Represents the column mean of the sample point set, and n represents the number of columns of the sample matrix;

[0027] As a preferred technical solution, the formula for calculating each element of the covariance matrix is:

[0028]

[0029] Among them, r ij is the element in row i and column j of the covariance matrix, n represents the number of columns in the standardized matrix, S ki Represents the element in the kth row and ith column of the normalized matrix, S kj Represents the element at row k and column j in the normalized matrix.

[0030] As a preferred technical solution, the S2 is specifically:

[0031] Convert the Plücker coordinates of the first arrow in the world coordinate system to the Plücker coordinates in the arrow coordinate system, and extract the 3D feature line of the first arrow in the arrow coordinate system as a comparison object;

[0032] Extracting the 2D feature line of the second arrow in the current scene image and performing 3D-2D matching with the comparison object;

[0033] The current position of the second arrow is calculated based on the EPnL algorithm, and the three-dimensional model of the second arrow is restored.

[0034] According to a second aspect of the present invention, there is provided a device for optimizing arrow recognition, the device comprising:

[0035] The first module is used for preprocessing the first arrow, which refers to all arrows of different specifications that appear for the first time in the recognition process. The captured scene image is segmented to obtain each object in the scene image, and the object name and rectangular bounding box attributes are set; objects with the name attribute of arrow are selected as objects to be processed, the initial feature lines in the rectangular bounding box of the object to be processed are extracted, and semantic information is assigned to the feature lines; the semantic information is the arrow ID;

[0036] The second module is used for processing the initial feature lines of the first arrow, screening multiple feature lines with the same arrow ID based on semantic information, converting the multiple feature lines into 3D to obtain multiple 3D feature lines; and performing point sampling on the multiple 3D feature lines to obtain a sample point set in three-dimensional space;

[0037] The third module is used to reconstruct the first arrow, calculate the main direction and the secondary direction of the arrow based on the sample point set in the three-dimensional space using the PCA algorithm; establish a new coordinate system in the world coordinate system based on the main and secondary directions, restore the arrow pose in this coordinate system, realize the reconstruction of the first arrow, and save the reconstruction result;

[0038] The fourth module is used for second arrow recognition. The second arrow refers to the arrow with the same specifications as the first arrow that appears repeatedly during the recognition process. The recognition of the second arrow is simplified based on the first arrow. The specific steps are as follows: based on the reconstructed first arrow, a comparison object is obtained, the 2D feature line of the second arrow is extracted and matched with the comparison object, and the EPnL algorithm is used to restore the position of the second arrow based on the matching result.

[0039] According to a third aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method when executing the program.

[0040] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] 1) The first arrow captured in the scene is used as a template. When arrows of the same specifications are subsequently identified, there is no need to rebuild them. Instead, they are compared with the template arrow, which simplifies the arrow recognition process and improves the arrow recognition rate.

[0043] 2) This method does not rely on learning models to detect and reconstruct arrows. It is applicable to different underground garage scenarios and has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of a method for optimizing arrow recognition according to the present invention;

[0045] Figure 2 Schematic diagram of the segmentation result of the present invention;

[0046] Figure 3 This is a schematic diagram of 3D feature line sampling in the present invention;

[0047] Figure 4 This is a schematic diagram of the main and secondary directions of arrows in the present invention;

[0048] Figure 5 The first arrow is a schematic diagram of PCA plane fitting and optimization of the present invention;

[0049] Figure 6 This is the reconstruction effect diagram of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0051] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0052] Example 1:

[0053] This embodiment provides a method for optimizing arrow recognition, and its process is as follows: Figure 1 As shown, the specific steps are:

[0054] S1: Identify and save the first arrow, where the first arrow refers to all arrows of different specifications that appear for the first time during the identification process. The specific steps are as follows:

[0055] S11, pre-processing: Use the SAM model to segment the captured image, obtain the objects in the scene image, and set the object name and rectangular bounding box attributes; filter the objects with the name attribute of arrow as the objects to be processed, extract the initial feature lines in the rectangular bounding box of the objects to be processed, and assign semantic information to the feature lines; the semantic information is the arrow ID; the effect after segmentation is as follows Figure 2 As shown;

[0056] S12, initial feature line processing: based on the semantic information, filter out three feature lines with the same arrow ID, add the arrow and semantic information to the three feature lines with the corresponding arrow ID as binary edges, and obtain three 3D feature lines; perform point sampling on the three 3D feature lines to obtain a sample point set in three-dimensional space. The sampling situation is as follows: Figure 3 As shown, the red solid line part in the figure is sampled;

[0057] S13. First arrow reconstruction specifically includes:

[0058] S131, constructing a sample matrix according to the sample point set and calculating column means, obtaining a normalized matrix based on the column means, processing the normalized matrix to obtain a covariance matrix, obtaining eigenvectors and eigenvalues ​​of the covariance matrix, and sorting the eigenvalues;

[0059] The formula for calculating the column mean is:

[0060]

[0061] in represents the column mean of the sample point set, n represents the number of columns of the sample matrix, p ij Represents the element in row i and column j of the sample matrix;

[0062] The formula for calculating each element of the normalized matrix is:

[0063]

[0064] Among them S ij represents the element in row i and column j of the normalized matrix, p ij represents the element in row i and column j of the sample matrix, Represents the column mean of the sample point set, and n represents the number of columns of the sample matrix;

[0065] The formula for calculating each element of the covariance matrix is:

[0066]

[0067] Among them, r ij is the element in row i and column j of the covariance matrix, n represents the number of columns in the standardized matrix, S ki Represents the element in the kth row and ith column of the normalized matrix, S kj Represents the element at row k and column j in the normalized matrix.

[0068] S132. Select the eigenvector direction corresponding to the largest eigenvalue as the main direction, and the eigenvector direction corresponding to the second largest eigenvalue as the secondary direction;

[0069] S133, normalize the main and secondary directions to obtain the main vector and the secondary vector, the main vector is set as the x-axis, the secondary vector is set as the y-axis, the result vector of the main vector cross-multiplied by the secondary vector is used as the z-axis, calculate the mean of the sample points, and use this value as the coordinate origin. Figure 4 As shown, the direction indicated by the red arrow is the main direction, and the direction indicated by the green arrow is the secondary direction.

[0070] After step S133, a schematic diagram of the first arrow after plane fitting and optimization based on the PCA algorithm can be obtained, as shown in FIG. Figure 5 shown.

[0071] S2: Simplify the identification of the second arrow based on the first arrow. The second arrow refers to the arrow with the same specifications as the first arrow that appears repeatedly during the identification process, specifically including:

[0072] S21. Converting the Plücker coordinates of the first arrow in the world coordinate system into the Plücker coordinates in the arrow coordinate system, and extracting the 3D feature line of the first arrow in the arrow coordinate system as a comparison object;

[0073] S22, extracting the 2D feature line of the second arrow in the current scene image and performing 3D-2D matching with the comparison object;

[0074] S23. Calculate the current position and posture of the second arrow based on the EPnL algorithm, and restore the three-dimensional model of the second arrow.

[0075] The process of calculating the pose of the second arrow based on the EPnL algorithm in step S23 can be found in the relevant process in the article "EPnL: An Efficient and Accurate Algorithm for Solving PnL Problems", which will not be repeated in this embodiment.

[0076] The above is an introduction to a method embodiment. The following further illustrates the solution of the present invention through an apparatus embodiment.

[0077] Example 2:

[0078] This embodiment provides a device for arrow optimization recognition, which is used to implement the above method. The device is placed on the car and starts working when entering the underground garage. The workflow of each module of the device refers to the method flow in Example 1, and finally the display terminal displays the following: Figure 6 The effect shown is that the arrow is red, the green is the road, and the purple is other things in the scene.

[0079] The present embodiment also provides an electronic device, including a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0080] Many components in a device are connected to the I / O interface, including: input units, such as a keyboard and mouse; output units, such as various types of displays and speakers; storage units, such as magnetic disks and optical disks; and communication units, such as network cards, modems, and wireless communication transceivers. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.

[0081] The processing unit performs the various methods and processes described above, such as methods S1 to S2. For example, in some embodiments, methods S1 to S12 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S2 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S2 in any other appropriate manner (e.g., by means of firmware).

[0082] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0083] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0084] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0085] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0086] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. An arrow optimization recognition method, characterized in that: The method includes: S1: Identify and save the first arrow. The first arrow refers to all arrows of different sizes that appear for the first time during the identification process. The specific steps are as follows: Preprocessing: Segment the captured scene image to obtain each object in the scene image, and set the object name and rectangular bounding box attributes; filter objects with arrow name attributes as objects to be processed, extract the initial feature lines in the rectangular bounding box of the object to be processed, and assign semantic information to the feature lines; the semantic information is the arrow ID; Initial feature line processing: multiple feature lines with the same arrow ID are screened out based on semantic information, and the multiple feature lines are converted into 3D to obtain multiple 3D feature lines; point sampling is performed on the multiple 3D feature lines to obtain a sample point set in three-dimensional space; First arrow reconstruction: Based on the sample point set in the three-dimensional space, the main direction and the secondary direction of the arrow are calculated using the PCA algorithm; a new coordinate system is established in the world coordinate system based on the main and secondary directions, and the arrow position is restored in this coordinate system to achieve reconstruction of the first arrow and save the reconstruction result. Specifically: According to the sample point set, a sample matrix is ​​constructed to calculate the column mean. A standardized matrix is ​​obtained based on the column mean. The standardized matrix is ​​processed to obtain a covariance matrix. The eigenvectors and eigenvalues ​​of the covariance matrix are obtained, and the eigenvalues ​​are sorted. The formula for calculating the column mean is: in represents the column mean of the sample point set, n represents the number of columns of the sample matrix, p ij Represents the element in row i and column j of the sample matrix; The formula for calculating each element of the normalized matrix is: Among them S ij represents the element in row i and column j of the normalized matrix, p ij represents the element in row i and column j of the sample matrix, Represents the column mean of the sample point set, and n represents the number of columns of the sample matrix; The formula for calculating each element of the covariance matrix is: Among them, r ij is the element in row i and column j of the covariance matrix, n represents the number of columns in the standardized matrix, S ki Represents the element in the kth row and ith column of the normalized matrix, S kj Represents the element in the kth row and jth column of the normalized matrix; The eigenvector direction corresponding to the largest eigenvalue is selected as the main direction, and the eigenvector direction corresponding to the second largest eigenvalue is selected as the secondary direction; Normalize the main and secondary directions to obtain the main vector and the secondary vector. The main vector is set as the x-axis, the secondary vector is set as the y-axis, and the result vector of the main vector cross-multiplied by the secondary vector is used as the z-axis. Calculate the mean of the sample points and use this value as the coordinate origin. S2: Simplify the identification of the second arrow based on the first arrow. The second arrow refers to the arrow with the same specifications as the first arrow that appears repeatedly during the identification process. The specific steps are as follows: a comparison object is obtained based on the reconstructed first arrow, the 2D feature line of the second arrow is extracted and matched with the comparison object, and the EPnL algorithm is used to restore the position of the second arrow according to the matching result.

2. The arrow optimization recognition method according to claim 1, characterized in that: In the preprocessing step, the SAM model is used to segment the collected image.

3. The arrow optimization recognition method according to claim 1, characterized in that: The method for realizing 3D conversion of the initial feature lines is as follows: adding arrows and semantic information as binary edges of multiple feature lines corresponding to arrow IDs to obtain multiple 3D feature lines.

4. The arrow optimization recognition method according to claim 3, characterized in that: The number of the characteristic lines is at least three.

5. The arrow optimization recognition method according to claim 1, characterized in that: The S2 is specifically: Convert the Plücker coordinates of the first arrow in the world coordinate system to the Plücker coordinates in the arrow coordinate system, and extract the 3D feature line of the first arrow in the arrow coordinate system as a comparison object; Extracting the 2D feature line of the second arrow in the current scene image and performing 3D-2D matching with the comparison object; The current position of the second arrow is calculated based on the EPnL algorithm, and the three-dimensional model of the second arrow is restored.

6. An arrow optimization recognition device, characterized in that: The device is used to implement the method according to any one of claims 1 to 5, comprising: The first module is used for preprocessing the first arrow, which refers to all arrows of different specifications that appear for the first time in the recognition process. The captured scene image is segmented to obtain each object in the scene image, and the object name and rectangular bounding box attributes are set; objects with the name attribute of arrow are selected as objects to be processed, the initial feature lines in the rectangular bounding box of the object to be processed are extracted, and semantic information is assigned to the feature lines; the semantic information is the arrow ID; The second module is used for processing the initial feature lines of the first arrow, screening multiple feature lines with the same arrow ID based on semantic information, converting the multiple feature lines into 3D to obtain multiple 3D feature lines; and performing point sampling on the multiple 3D feature lines to obtain a sample point set in three-dimensional space; The third module is used to reconstruct the first arrow, calculate the main direction and the secondary direction of the arrow based on the sample point set in the three-dimensional space using the PCA algorithm; establish a new coordinate system in the world coordinate system based on the main and secondary directions, restore the arrow pose in this coordinate system, realize the reconstruction of the first arrow, and save the reconstruction result; The fourth module is used for second arrow recognition. The second arrow refers to the arrow with the same specifications as the first arrow that appears repeatedly during the recognition process. The recognition of the second arrow is simplified based on the first arrow. The specific steps are as follows: based on the reconstructed first arrow, a comparison object is obtained, the 2D feature line of the second arrow is extracted and matched with the comparison object, and the EPnL algorithm is used to restore the position of the second arrow based on the matching result.

7. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images