Gate recognition method and device, electronic equipment, storage medium and product

Abstract

The invention relates to a gate recognition method and device, electronic equipment, a storage medium and a product. The gate recognition method comprises the following steps: acquiring a to-be-recognized gate image acquired by image acquisition equipment; identifying a plurality of key position points in the to-be-identified gate image through the identification network model to generate a plurality of key point heat maps, the key point heat maps comprising a plurality of heat areas, each heat area having a corresponding heat mark, and the heat marks being used for indicating the probability that the corresponding heat area contains the key position points; obtaining key points corresponding to the plurality of key position points according to the plurality of key point heat maps; and identifying the gate under the condition that the position relationship among the plurality of key points meets the preset position relationship. According to the identification network model, the gate machine can be identified according to the position relationship among the plurality of key points only by identifying the plurality of key points, so that the model calculation amount is effectively reduced, and the gate machine identification accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent vehicle technology, and specifically to a gate recognition method, device, electronic equipment, storage medium and product. Background Art

[0002] In the field of intelligent vehicle driving, image processing systems can detect and identify targets such as vehicles, pedestrians, obstacles, road signs, traffic signals, etc., and are widely used in visual perception and environmental understanding.

[0003] As a type of obstacle, the detection of turnstiles is an important part of the vehicle's environmental understanding. At present, turnstiles are mainly identified and understood through image processing and deep learning technology. However, there are still problems such as large amount of calculation and large recognition error. Summary of the invention

[0004] One of the purposes of the present application is to provide a gate recognition method to solve the problem of large amount of calculation and large recognition error in the process of gate recognition using image processing and deep learning technology in the related art. The second purpose is to provide a gate recognition device. The third purpose is to provide an electronic device. The fourth purpose is to provide a computer-readable storage medium. The fifth purpose is to provide a computer program product.

[0005] In order to achieve the above purpose, the technical solution adopted in this application is as follows:

[0006] In a first aspect, the present application provides a gate recognition method, which includes: obtaining a gate image to be identified acquired by an image acquisition device; identifying multiple key position points in the gate image to be identified through an identification network model to generate multiple key point heat maps, wherein the key point heat map includes multiple heat areas, each heat area has a corresponding heat mark, and the heat mark is used to indicate the probability that the corresponding heat area contains the key position point; the multiple key position points include at least one of the gate wing starting point, gate wing midpoint and gate wing end point of the gate; according to the multiple key point heat maps, key points corresponding to the multiple key position points are obtained, wherein one key point heat map obtains a key point corresponding to one key position point; when the positional relationship between the multiple key points meets the preset positional relationship, the gate is identified.

[0007] According to the above technical means, by inputting the obtained gate image to be identified into the recognition network model, multiple key point heat maps can be obtained, and then the key points corresponding to the multiple key position points on the gate can be obtained. Then, the gate can be identified based on the multiple key points when the positional relationship between the multiple key points meets the preset positional relationship. In this process, the recognition network model only needs to identify multiple key points, and the gate can be identified based on the positional relationship between the multiple key points, which effectively reduces the model calculation amount and improves the accuracy of gate identification.

[0008] Furthermore, when the positional relationship between multiple key points satisfies a preset positional relationship, the gate is identified, including: obtaining a first key point from multiple key points; based on the first key point, determining multiple second key points from the multiple key points, wherein a first key position point corresponding to the first key point and a second key position point corresponding to the second key point have a corresponding relationship; the first key position point and the second key position point are included in the key position point; when the positional relationship between the first key point and the multiple second key points satisfies the preset positional relationship, the gate is identified.

[0009] According to the above technical means, by obtaining a first key point and multiple second key points corresponding to the first key point from multiple key points, the gate can be identified based on specific key position points on the gate to meet the user's usage needs.

[0010] Further, when the first key point corresponds to the midpoint of the gate wing, when the positional relationship between the first key point and multiple second key points satisfies a preset positional relationship, the gate is identified, including: determining a third key point and a fourth key point from multiple second key points, the distance between the third key position point corresponding to the third key point and the first key position point is equal to the distance between the fourth key position point corresponding to the fourth key point and the first key position point, the third key position point and the fourth key position point are located on both sides of the first key position point; the third key position point and the fourth key position point are included in the second key position point; determining a first distance between the first key point and the third key point, and a second distance between the first key point and the fourth key point; and identifying the gate when the difference between the first distance and the second distance is less than a preset threshold.

[0011] According to the above technical means, when the first key point corresponds to the midpoint of the gate wing, two key points on both sides of the midpoint of the gate wing and at the same distance from the midpoint of the gate wing can be obtained, so that the gate can be identified based on the distance between the two key points and the midpoint of the gate wing. In this process, the positional relationship between the first key point and the third key point and the fourth key point is fully considered to achieve simple and efficient identification of the gate.

[0012] Further, when the first key point corresponds to the starting point or the end point of the gate wing, the gate is identified when the positional relationship between the first key point and multiple second key points satisfies a preset positional relationship, including: determining the orientation of each second key point among the multiple second key points relative to the first key point, wherein each second key point corresponds to an orientation; and identifying the gate when the orientation difference between the multiple orientations is less than a preset threshold.

[0013] According to the above technical means, when the first key point corresponds to the gate wing starting point or gate wing end point, the position of the second key point relative to the gate wing starting point or gate wing end point can be obtained, so that the gate can be identified when the positions of multiple second key points are similar. In this process, the positional relationship between the first key point and the second key point is fully considered to achieve simple and efficient identification of the gate.

[0014] Furthermore, when the multiple key position points include the gate wing starting point and the gate wing end point of the gate, after identifying the gate, the method also includes: obtaining a fifth key point corresponding to the gate wing starting point and a sixth key point corresponding to the gate wing end point from the multiple key points; determining the angle between the first direction formed by the fifth key point and the sixth key point and the horizontal direction; when the degree of the angle is greater than a preset threshold, determining that the gate wing is in an open posture.

[0015] According to the above technical means, whether the gate wing of the gate machine is open is determined by using the fifth key point corresponding to the starting point of the gate wing and the sixth key point corresponding to the end point of the gate wing among multiple key points, thereby meeting the user's identification needs for the gate machine.

[0016] Furthermore, based on multiple key point heat maps, key points corresponding to multiple key position points are obtained, including: determining the key position point corresponding to each key point heat map according to the output channel of each key point heat map in the multiple key point heat maps; determining the thermal value of each heat area in each key point heat map according to the heat mark; and determining the heat area with a thermal value greater than a preset threshold as the key point corresponding to the key position point.

[0017] According to the above technical means, the recognition network model can output a key point heat map, so that after determining the key point heat map, the key points corresponding to the key position points can be determined according to the thermal markers. In this process, the key points can be intuitively obtained through the key point heat map, which has the characteristics of high precision and strong flexibility.

[0018] Furthermore, before identifying multiple key position points in the gate image to be identified through the recognition network model, the method also includes: obtaining a labeled gate image; inputting the labeled gate image into the initial recognition model to obtain first output data and second output data; determining the loss function value of the initial recognition model based on the first output data and the second output data; based on the loss function value, optimizing the model parameters of the initial recognition model to obtain the recognition network model; wherein the first output data is used to indicate the heat marks of multiple heat areas, and the second output data is used to indicate the position codes of multiple heat areas.

[0019] According to the above technical means, the first output layer of the initial recognition model can output the first output data, and the second output layer can output the second output data of the key points. Based on the first output data, the second output data of the key points and the labeled gate image input into the initial recognition model, the loss of the initial recognition model can be determined, thereby optimizing the initial recognition model and obtaining the recognition network model.

[0020] In the second aspect, the present application also provides a gate recognition device, which includes: a first processing module, used to obtain a gate image to be identified captured by an image acquisition device; a second processing module, used for the second processing module, used to identify multiple key position points in the gate image to be identified through an identification network model to generate multiple key point heat maps, wherein the key point heat map includes multiple heat areas, each heat area has a corresponding heat mark, and the heat mark is used to indicate the probability that the corresponding heat area contains the key position point; the multiple key position points include at least one of the gate wing starting point, gate wing midpoint and gate wing end point of the gate; a third processing module, used to obtain key points corresponding to multiple key position points according to multiple key point heat maps, wherein one key point heat map obtains a key point corresponding to one key position point; a fourth processing module, used to identify the gate when the positional relationship between the multiple key points meets the preset positional relationship.

[0021] In a third aspect, the present application further provides an electronic device, the electronic device comprising a memory and one or more processors, the memory storing a computer program, and when the computer program is executed by the electronic device, the gate identification method as described in any one of the first aspects is implemented.

[0022] In a fourth aspect, the present application further provides a computer-readable storage medium, which stores executable instructions, wherein when the executable instructions are executed by a processor, the gate identification method as described in any one of the first aspects is implemented.

[0023] In a fifth aspect, the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the gate identification method as described in any one of the first aspects.

[0024] It should be noted that the technical effects of the third to fifth aspects can refer to the detailed description of the first and second aspects mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of a first optional flow chart of a gate identification method provided in an embodiment of the present application;

[0026] Figure 2 A second optional flow chart of the gate identification method provided in the embodiment of the present application;

[0027] Figure 3 A third optional flow chart of the gate identification method provided in the embodiment of the present application;

[0028] Figure 4 A fourth optional flow chart of the gate identification method provided in the embodiment of the present application;

[0029] Figure 5 A fifth optional flow chart of the gate identification method provided in the embodiment of the present application;

[0030] Figure 6 A sixth optional flow chart of the gate identification method provided in the embodiment of the present application;

[0031] Figure 7 A seventh optional flow chart of the gate identification method provided in the embodiment of the present application;

[0032] Figure 8 A first optional schematic diagram of a marked gate image provided in an embodiment of the present application;

[0033] Fig. 9 A second optional schematic diagram of a marked gate image provided in an embodiment of the present application;

[0034] Fig.10 An optional schematic diagram of first output data provided in an embodiment of the present application;

[0035] Fig.11 A first optional schematic diagram of the key points provided in the embodiments of the present application;

[0036] Fig.12 A second optional schematic diagram of the key points provided in the embodiments of the present application;

[0037] Fig.13 An optional schematic diagram of a gate identification device provided in an embodiment of the present application;

[0038] Fig.14An optional structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0040] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0041] In a first aspect, the present application provides a gate identification method. Figure 1 A first optional flow chart of the gate identification method provided in the embodiment of the present application is as follows: Figure 1 As shown, the gate identification method may include:

[0042] Step S101, obtaining the gate image to be identified captured by the image acquisition device.

[0043] It is understandable that the vehicle is provided with an image acquisition device, such as a vehicle-mounted camera. The image acquisition device can acquire an image of the area where the gate to be identified is located to obtain an image of the gate to be identified. The electronic device can obtain the image of the gate to be identified through the image acquisition device for subsequent identification of the gate image to be identified.

[0044] In some embodiments, a gate, also known as a channel gate, access control gate, etc., is an automated device used to control the passage of people or vehicles. It is widely used in various places where access control is required, such as subway stations, railway stations, etc. The main function of the gate is to manage the passing personnel and ensure that only authorized personnel or vehicles can pass.

[0045] In some embodiments, the gate can be composed of a base and gate wings. According to the difference in gate wings, the gate can be divided into multiple types of gates such as single-pole gates and double-pole gates. In the embodiments of the present application, at least the identification of the above-mentioned single-pole gates and double-pole gates can be realized.

[0046] Step S102, identifying a plurality of key position points in the gate image to be identified by using a recognition network model, so as to generate a plurality of key point heat maps.

[0047] It can be understood that the electronic device can input the gate image to be identified into the recognition network model, and the recognition network model can identify multiple key position points in the gate image to be identified to generate multiple key point heat maps.

[0048] In some embodiments, key position points may refer to positions on the gate machine that are helpful for gate machine identification. Key position points are preset positions that need to be identified by the network model for identification. These position points are usually located on the gate wing of the gate machine. In one embodiment, the multiple key position points include at least one of the gate wing starting point, the gate wing midpoint and the gate wing end point of the gate machine. In one embodiment, the multiple key points also include the gate wing quarter-division point, the gate wing tri-division point, etc.

[0049] It can be understood that the gate wing starting point of the gate machine can be located at the position where the gate wing is connected to the base; the gate wing end point of the gate machine can be located at the position where the gate wing is away from the base; the gate wing midpoint of the gate machine can be located in the middle of the gate wing starting point and the gate wing end point. The gate wing quarter point can be located at a position with a quarter of the gate wing length from the gate wing starting point or the gate wing end point. The gate wing trisection point can be located at a position with a third of the gate wing length from the gate wing starting point or the gate wing end point.

[0050] In some embodiments, the gate wing has a certain thickness, so that the connection between the gate wing and the base is longer. In this case, the gate wing starting point, the gate wing midpoint and the gate wing end point of the gate machine can all be located at the midpoint of the gate wing in the thickness direction. Exemplarily, the gate wing starting point of the gate machine can be located at the midpoint of the connection line between the gate wing and the base.

[0051] It can be understood that the shape of the gate wing in the gate image to be identified can be a rectangle. By obtaining the starting point, midpoint and end point of the gate wing, the characteristics of the rectangle in the length direction can be obtained, which is helpful for identifying the gate.

[0052] It can be understood that the recognition network model can output multiple key point heat maps corresponding to the recognition gate image through multiple output channels. Here, each output channel of the recognition network model can output a key point heat map. Each output channel of the recognition network model corresponds to outputting a key point heat map of a key position point.

[0053] In some embodiments, a key point heat map is a technique for visualizing the probability distribution of key point locations. In key point detection, the method of forming a key point heat map is actually to perform feature matching in the spatial dimension, by "sliding" the convolution kernel on the feature map plane, focusing on and utilizing local information, and thus locating the key points of feature matching.

[0054] In some embodiments, the key point heat map includes multiple heat areas, each heat area has a corresponding heat mark, and the heat mark is used to indicate the probability that the corresponding heat area contains the key position point.

[0055] It is understandable that the key point heat map may include multiple heat areas, each heat area corresponds to a pixel area of ​​the gate image to be identified. Here, each area on the key point heat map has a corresponding heat mark, which can be used to indicate the probability that the corresponding heat area contains a key position point.

[0056] In some embodiments, the heat mark may be a numerical value, such as a numerical value of 1, a numerical value of 2, etc. The larger the numerical value, the greater the probability that the heat area contains a key location point.

[0057] Step S103, obtaining key points corresponding to multiple key position points according to multiple key point heat maps.

[0058] It is understandable that after obtaining multiple key point heat maps, the electronic device can obtain key points corresponding to multiple key position points on the gate for each key point heat map in the multiple key point heat maps. Here, one key point heat map can correspond to one key point corresponding to one key position point.

[0059] It can be understood that the key point is the position determined according to the key point heat map, that is, the key point is identified by the recognition network model. Key points are different from key position points. Key position points are fixed positions, while key points are not fixed positions. When the recognition network model has high recognition accuracy, the key point and the key position point can indicate the same pixel point on the gate image to be identified; when the recognition network model has low recognition accuracy, the key point will deviate and cannot indicate the same pixel point on the gate image to be identified as the key position point.

[0060] In some embodiments, based on the heat mark of each heat area in the key point heat map, the heat area with the highest probability of containing the key position point in the key point heat map is determined, and the point corresponding to the heat area on the gate image to be identified is determined as the key point.

[0061] In some embodiments, obtaining key points corresponding to a plurality of key position points includes: obtaining coordinate information of the plurality of key points.

[0062] It is understandable that, according to the key point heat map, the position information of any heat area can be obtained, and then the coordinate information of the key point can be determined according to the position information of any heat area. After obtaining the coordinate information of the key point, the gate can be subsequently identified by the position of the key point.

[0063] Step 104 is to identify the gate when the positional relationship between the multiple key points meets the preset positional relationship.

[0064] It is understandable that after obtaining multiple key points, the electronic device can obtain information of the multiple key points, and then determine whether the positional relationship between the multiple key points satisfies the preset positional relationship. In the case where the positional relationship between the multiple key points satisfies the preset positional relationship, the gate can be identified in the gate image to be identified. In the case where the positional relationship between the multiple key points does not satisfy the preset positional relationship, the gate cannot be identified in the gate image to be identified, that is, the gate image to be identified does not include the gate.

[0065] It is understandable that the electronic device can obtain key position points corresponding to multiple key points, as well as the coordinate information of the key position points in the image to be identified. Subsequently, the electronic device can identify the gate based on the multiple key position points and the positions of the key position points. Since the multiple position key points have a certain positional relationship on the gate, for example, the midpoint of the gate wing is located between the starting point and the end point of the gate wing, therefore, when the positional relationship between the multiple key position points meets the preset positional relationship, the electronic device can identify the gate.

[0066] In the embodiment of the present application, by inputting the obtained gate image to be identified into the recognition network model, multiple key point heat maps can be obtained, and then the key points corresponding to the multiple key position points on the gate can be obtained, and then the gate can be identified based on the multiple key points when the positional relationship between the multiple key points meets the preset positional relationship. In this process, the recognition network model only needs to identify multiple key points, and the gate can be identified according to the positional relationship between the multiple key points, which effectively reduces the model calculation amount and improves the accuracy of gate identification.

[0067] In some possible implementations, Figure 2 A second optional flow chart of the gate identification method provided in the embodiment of the present application is as follows: Figure 2 As shown, step S104 may include:

[0068] Step S201, obtaining a first key point from multiple key points.

[0069] It is understandable that the electronic device can obtain any one key point from the multiple key points and determine the key point as the first key point. After obtaining the first key point, the electronic device can determine the key position point corresponding to the first key point.

[0070] Step S202: Based on the first key point, determine a plurality of second key points from the plurality of key points.

[0071] It can be understood that after obtaining the first key point, the electronic device can determine the first key position point corresponding to the first key point, and then determine multiple second key points corresponding to the multiple second key position points from the multiple key points based on the multiple second key position points corresponding to the first key position point.

[0072] In some embodiments, the correspondence between the first key position point and the second key position point can be set according to actual needs. For example, when the first key position point is the starting point of the gate wing, the multiple second key position points corresponding to the first key position point can be the gate wing end point and the gate wing midpoint; the multiple second key position points corresponding to the first key position point can also be the gate wing trisection point and the gate wing end point.

[0073] Step S203, identifying the gate when the positional relationship between the first key point and the plurality of second key points satisfies a preset positional relationship.

[0074] It can be understood that after the electronic device determines the first key point and multiple second key points corresponding to the first key point, the electronic device can obtain the positional relationship between the first key point and the multiple second key points. Subsequently, the electronic device can determine whether the positional relationship between the first key point and the multiple second key points satisfies the preset positional relationship, and identify the gate if the positional relationship between the first key point and the multiple second key points satisfies the preset positional relationship.

[0075] It can be understood that the electronic device can obtain the positional relationship between the first key point and the multiple second key points through information such as the coordinates of the first key point and the multiple second key points, the distance between the first key point and the multiple second key points, etc. According to the positional relationship between the first key point and the multiple second key points, the arrangement of the first key point and the multiple second key points on the gate image to be identified can be obtained, so that when the arrangement of the above key points is consistent with the arrangement of the key position points on the gate, it can be determined that the positional relationship between the first key point and the multiple second key points meets the preset positional relationship, thereby identifying the gate.

[0076] Exemplarily, when the first key point corresponds to the gate wing starting point, and the multiple second key points correspond to the gate wing trisection point and the gate wing end point, respectively, and the gate wing trisection point is located between the gate wing starting point and the gate wing end point, it can be determined that the positional relationship between the first key point and the multiple second key points meets the preset positional relationship. When the first key point corresponds to the gate wing starting point, and the multiple second key points correspond to the gate wing midpoint and the gate wing end point, respectively, and the distances from the gate wing midpoint to the gate wing starting point and the gate wing end point are similar, it can be determined that the positional relationship between the first key point and the multiple second key points meets the preset positional relationship.

[0077] In some embodiments, the first key point has a positional relationship with each of the multiple second key points, so that multiple positional relationships can be obtained. By comparing the multiple positional relationships, it can be determined whether the positional relationship between the first key point and the multiple second key points satisfies the preset positional relationship.

[0078] It is understandable that the electronic device can obtain the positional relationship between the first key point and each second key point through the coordinates, distances, and other information of the plurality of second key points relative to the first key point. Subsequently, the electronic device can determine whether the positional relationship between the first key point and the plurality of second key points satisfies the preset positional relationship by comparing the position of each second key point relative to the first key point.

[0079] In an embodiment of the present application, by obtaining a first key point and multiple second key points corresponding to the first key point from multiple key points, the gate can be identified based on specific key position points on the gate to meet the user's usage needs.

[0080] In some possible implementations, Figure 3 A third optional flow chart of the gate identification method provided in the embodiment of the present application is as follows: Figure 3 As shown, when the first key point corresponds to the midpoint of the gate wing, step S203 may include:

[0081] Step S301, determining a third key point and a fourth key point from a plurality of second key points.

[0082] It can be understood that when the first key point is the midpoint of the gate wing, the electronic device can determine the third key point and the fourth key point from multiple second key points corresponding to the first key point.

[0083] In some embodiments, the third key point corresponds to a third key position point on the gate. Since the third key point is included in the second key point, the third key position point is included in the second key position point corresponding to the second key point. The fourth key point corresponds to a fourth key position point on the gate. Since the fourth key point is included in the second key point, the fourth key position point is also included in the second key position point corresponding to the second key point.

[0084] In some embodiments, the distance between the third key position point corresponding to the third key point and the first key position point corresponding to the first key point is equal to the distance between the fourth key position point corresponding to the fourth key point and the first key position point corresponding to the first key point. And the third key position point and the fourth key position point are respectively located on both sides of the first key position point. In other words, when the first key point corresponds to the midpoint of the gate wing, the third key position point and the fourth key position point are respectively located on both sides of the midpoint of the gate wing, and the third key position point and the fourth key position point are at the same distance from the midpoint of the gate wing.

[0085] Exemplarily, when the first key point corresponds to the midpoint of the gate wing, the third key point can correspond to the starting point of the gate wing, and the fourth key point can correspond to the end point of the gate wing; or, when the first key point corresponds to the midpoint of the gate wing, the third key point and the fourth key point can respectively correspond to the two quarter-division points of the gate wing on both sides of the midpoint of the gate wing.

[0086] Step S302: determine a first distance between the first key point and the third key point, and a second distance between the first key point and the fourth key point.

[0087] It can be understood that after obtaining the third key point and the fourth key point, the electronic device can determine the first distance between the first key point and the third key point and the second distance between the first key point and the fourth key point.

[0088] Step S303: When the difference between the first distance and the second distance is less than a preset threshold, identify the gate.

[0089] It can be understood that the electronic device can determine that the distances from the third key point and the fourth key point to the first key point are similar when the difference between the first distance and the second distance is less than a preset threshold, thereby identifying the gate.

[0090] It can be understood that the third key point corresponding to the third key point and the fourth key point corresponding to the fourth key point are respectively located on both sides of the first key point and have the same distance from the first key point. Therefore, after identifying the third key point and the fourth key point determined by the network model, when the electronic device determines that the difference between the distance between the third key point and the fourth key point and the first key point is less than a preset threshold, the gate can be identified according to multiple key points.

[0091] In the embodiment of the present application, when the first key point corresponds to the midpoint of the gate wing, two key points on both sides of the midpoint of the gate wing and at the same distance from the midpoint of the gate wing can be obtained, so that the gate can be identified based on the distance between the two key points and the midpoint of the gate wing. In this process, the positional relationship between the first key point and the third key point and the fourth key point is fully considered to achieve simple and efficient identification of the gate.

[0092] In some possible implementations, Figure 4 A fourth optional flow chart of the gate identification method provided in the embodiment of the present application is as follows: Figure 4 As shown, when the first key point corresponds to the gate wing starting point or gate wing end point, step S203 may include:

[0093] Step S401: determine the position of each second key point among a plurality of second key points relative to the first key point.

[0094] It can be understood that the electronic device can determine the position of each second key point relative to the first key point when the first key point corresponds to the starting point or the end point of the gate wing.

[0095] It can be understood that the electronic device can obtain the coordinate information of the first key point and the second key point. Based on the coordinates of the first key point and the second tube, a vector between the first key point and each second key point can be constructed, so that the position of each second key point corresponding to the first key point can be obtained. Each second key point corresponds to one position, and multiple second key points correspond to multiple positions.

[0096] Step S402, identifying the gate when the orientation difference between the multiple orientations is less than a preset threshold.

[0097] It can be understood that after obtaining multiple orientations, the electronic device can determine the orientation difference between the multiple orientations, and identify the gate when the orientation difference between the multiple orientations is less than a preset threshold.

[0098] It can be understood that when the first key point corresponds to the gate wing starting point or gate wing end point, the second key position point corresponding to the second key point is located on the side of the first key position point corresponding to the first key point. Based on this, the orientations of multiple second key points relative to the first key point should be similar, that is, the orientation difference between multiple orientations should be less than the preset threshold. Then, the electronic device can identify the gate when the orientation difference between multiple orientations is less than the preset threshold.

[0099] In some embodiments, the electronic device can identify the gate when the orientation difference between any two orientations among multiple orientations is less than a preset threshold.

[0100] In the embodiment of the present application, when the first key point corresponds to the gate wing starting point or gate wing end point, the position of the second key point relative to the gate wing starting point or gate wing end point can be obtained, so that the gate can be identified when the positions of multiple second key points are similar. In this process, the positional relationship between the first key point and the second key point is fully considered to achieve simple and efficient identification of the gate.

[0101] In some possible implementations, Figure 5 A fifth optional flow chart of the gate identification method provided in the embodiment of the present application is as follows: Figure 5 As shown, after step S104, the method further includes:

[0102] Step S501, obtaining the fifth key point corresponding to the starting point of the gate wing and the sixth key point corresponding to the end point of the gate wing from multiple key points.

[0103] It can be understood that when multiple key position points include the gate wing starting point and the gate wing end point of the gate machine, the electronic device can obtain the fifth key point corresponding to the gate wing starting point and the sixth key point corresponding to the gate wing end point from multiple key points corresponding to the multiple key position points.

[0104] It should be noted that when the gate has more than one gate wing, multiple gate wings can be identified. When determining the gate wing posture, the gate wing starting point and gate wing end point corresponding to the same gate wing can be obtained, so as to determine each gate wing.

[0105] Step S502: determine the angle between the first direction formed by the fifth key point and the sixth key point and the horizontal direction.

[0106] It is understandable that after obtaining the fifth key point and the sixth key point, the electronic device can determine the coordinates of the fifth key point and the sixth key point, and then determine the first direction of the line connecting the fifth key point and the sixth key point. After determining the first direction, the electronic device can determine the angle between the first direction and the horizontal direction.

[0107] Step S503, when the angle is greater than a preset threshold, determining that the gate wing is in an open posture.

[0108] It can be understood that after obtaining the angle between the first direction of the line connecting the fifth key point and the sixth key point and the horizontal direction, the electronic device can determine that the gate wing is in the open posture when the angle is greater than a preset threshold.

[0109] It can be understood that when the gate wing is in use, when the gate wing is in a closed position, the gate wing tail is flush with the gate wing head, and the angle between the first direction formed by the gate wing starting point and the gate wing end point and the horizontal direction is close to zero degrees; when the gate wing is in an open position, the gate wing tail is raised high, and the angle between the first direction formed by the gate wing starting point and the gate wing end point and the horizontal direction is much greater than zero degrees. Therefore, the electronic device can identify the gate wing as an open position when the degree of the angle is greater than a preset threshold; and identify the gate wing as a closed position when the degree of the angle is less than the preset threshold.

[0110] In some embodiments, the preset threshold may be 60 degrees.

[0111] In an embodiment of the present application, whether the gate wing of the gate machine is open can be determined based on the fifth key point corresponding to the starting point of the gate wing and the sixth key point corresponding to the end point of the gate wing among multiple key points, thereby meeting the user's identification needs for the gate machine.

[0112] In some possible implementations, Figure 6 A sixth optional flow chart of the gate identification method provided in the embodiment of the present application is as follows: Figure 6 As shown, step S103 may include:

[0113] Step 601, determining a key position point corresponding to each key point heat map according to an output channel of each key point heat map among a plurality of key point heat maps.

[0114] It can be understood that the electronic device can determine the key position point corresponding to each key point heat map on the gate by obtaining the output channel of each key point heat map.

[0115] In some embodiments, the recognition network model can perform multi-channel feature convolution, and different channels correspond to different key positions in the gate image to be recognized, thereby improving recognition efficiency. Similarly, the recognition network model can correspond to multiple output channels to output recognition results of different key positions.

[0116] Exemplarily, the first output channel of the recognition network model can output a key point heat map of the starting point of the gate wing, the second output channel can output a key point heat map of the midpoint of the gate wing, and the third output channel can output a key point heat map of the end point of the gate wing.

[0117] Step 602, determining the thermal value of each thermal area in each key point heat map according to the thermal mark.

[0118] It is understandable that each heat area in the key point heat map has a corresponding heat mark, and the heat mark is used to indicate the probability that the corresponding heat area contains the key position point. In some embodiments, the heat mark can be a thermal value, and the thermal value can be displayed in the form of a numerical value. The electronic device can determine the thermal value of each heat area through the heat mark of each heat area.

[0119] Step 603: determine the heat area whose thermal value is greater than the preset threshold as the key point corresponding to the key position point.

[0120] It can be understood that the electronic device can obtain a heat area whose thermal value is greater than a preset threshold, and determine the pixel area corresponding to the heat area in the gate image to be identified as a key point.

[0121] In some embodiments, each heat area in the key point heat map corresponds to a pixel area of ​​the gate image to be identified. Determining a heat area with a thermal value greater than a preset threshold as a key point corresponding to a key position point may include: determining a heat area with a thermal value greater than a preset threshold as a target heat area, determining a pixel area corresponding to the target heat area in the gate image to be identified according to the scaling ratio of the target heat area and the key point heat map, and determining the pixel area as a key point.

[0122] In the disclosed embodiment, the recognition network model can output a key point heat map, so that after determining the key point heat map, the key points corresponding to the key position points can be determined according to the thermal markers. In this process, the key points can be intuitively obtained through the key point heat map, which has the characteristics of high precision and strong flexibility.

[0123] In some possible implementation modes, Figure 7 A seventh optional flow chart of the gate identification method provided in the embodiment of the present application is as follows: Figure 7 As shown, before step S102, the gate recognition method further includes:

[0124] Step S701, obtaining a marked gate image.

[0125] It is understandable that after obtaining the image containing the gate collected by the vehicle, the image containing the gate can be manually annotated to obtain an annotated gate image. Subsequently, the electronic device can obtain one or more of the annotated gate images to train the model in a subsequent process.

[0126] In some embodiments, four key positions in the gate image have been marked, for example, the key positions may be the four endpoints of the gate wing (upper left, upper right, lower left and lower right), and the four endpoints may be marked. Figure 8 This is a first optional schematic diagram of a gate image with annotations provided in an embodiment of the present application. Figure 8 As shown, the four endpoints of the gate wing can be marked as endpoint 1, endpoint 2, endpoint 3 and endpoint 4. Among them, endpoint 1 and endpoint 2 are located at the connection between the gate wing and the base; endpoint 3 and endpoint 4 are located at the end of the gate wing away from the base.

[0127] In some embodiments, after obtaining the labeled gate image, the electronic device may construct a data iterator based on the labeled gate image.

[0128] It can be understood that the process of constructing a data iterator based on the labeled gate image can be selected according to actual needs, and the embodiment of the present application does not make any specific limitations on this.

[0129] Exemplarily, three key position points can be generated on each gate wing according to the four endpoints of the gate wing, and the three key position points are recorded as top point, middle point, and down point. Among them, the top point is the center point of the gate wing starting position close to the gate base, the middle point is the center point of the middle position of the gate wing, and the down point is the center point of the gate wing ending position away from the gate base. Based on the above three key position points, three heatmap matrices can be generated, each heatmap matrix represents the top point, the middle point, and the down point respectively.

[0130] For example, Fig. 9 This is a second optional schematic diagram of a gate image with annotations provided in an embodiment of the present application. Fig. 9 As shown, according to Figure 8 The four endpoints (endpoint 1, endpoint 2, endpoint 3 and endpoint 4) on the gate wing shown can obtain the top point, middle point and down point of the gate wing.

[0131] Taking the heat map matrix corresponding to the top point as an example, each element position (which can also be understood as a heat area) represents a pixel block (composed of multiple pixels) of size stride*stride in the annotated gate image. The heat map matrix is ​​used to indicate whether there is a top point of a gate wing at each element position in the matrix. If the annotated gate image contains the top point of the gate wing, the matrix value (which can also be understood as a heat mark) of the element position corresponding to the heat map matrix is ​​1, and the matrix values ​​of some element positions near the element position are non-zero values, and the distribution of the matrix values ​​of these element positions conforms to the Gaussian distribution.

[0132] In some embodiments, the heat map matrix can be calculated by formula (1).

[0133]

[0134] Among them, (p x ,p y ) is the element position corresponding to the top point in the annotated gate image in the heatmap matrix. xyz For the heat map matrix (p x ,p y ) matrix value.

[0135] It can be understood that the matrix value of the element position far away from the top point in the heatmap matrix is ​​0. Similarly, based on the above formula (1), the heatmap matrix corresponding to the middle point and the down point can also be obtained.

[0136] In some embodiments, during the process of constructing the data iterator, a first matrix and a second matrix may be created, wherein the first matrix and the second matrix respectively represent the distance from the top point to the upper left corner of the annotated gate image, and the distance from the down point to the upper left corner of the annotated gate image.

[0137] Step S702: input the labeled gate image into the initial recognition model to obtain first output data and second output data.

[0138] It can be understood that by inputting the labeled gate image into the initial recognition model, the initial recognition model can extract the image features of the labeled gate image, and then output the first output data and the second output data.

[0139] In some embodiments, the initial recognition model includes: a first output layer and a second output layer. The input data of the first output layer and the second output layer are the same, and the output data of the first output layer and the second output layer are different. Among them, the first output layer can be used to output first output data, and the first output data is used to indicate the heat marks of multiple heat areas. It can also be understood that the first output layer of the initial recognition model outputs a heat map matrix. The second output layer can be used to output second output data, and the second output data is used to indicate the position encoding of multiple heat areas. It can also be understood that the second output layer of the initial recognition model outputs the position encoding of key points. Both the first output data and the second output data are used to determine the model loss of the initial recognition model during the training process.

[0140] It can be understood that in the process of training the model parameters of the initial recognition model through the labeled gate image, the loss function of the initial recognition model can be constructed according to the first output data and the second output data output by the first output layer and the second output layer, so that the loss of the model can be determined by using the first output data and the second output data, and the adjustment direction of the model parameters can be determined, and the recognition network model can be obtained by repeating this cycle.

[0141] In some embodiments, the electronic device may obtain an initial recognition model and may also select a suitable convolutional network according to image recognition requirements to construct the initial recognition model.

[0142] It can be understood that after constructing the initial recognition model, the electronic device can input one or more labeled gate images into the initial recognition model for feature extraction. Subsequently, the first output layer of the initial recognition model can output the first output data corresponding to the one or more labeled gate images, and the second output layer of the initial recognition model can output the second output data of the key points corresponding to the one or more labeled gate images.

[0143] Exemplarily, the first output layer can output the heatmap matrix corresponding to the top point, the middle point, and the down point.

[0144] In some embodiments, the electronic device may select a variable group convolutional network (VARGNET) to construct an initial recognition model. The variable group convolutional network includes at least three network layers, namely a feature extraction layer, a connection layer, and an output layer. The output layer includes a first output layer and a second output layer, the first output layer is used to output first output data, and the second output layer is used to output second output data.

[0145] Here, the network structure and network parameters of the first output layer can be set according to actual needs, and the embodiments of the present application do not limit this.

[0146] Step S703: Determine the loss function value of the initial recognition model according to the first output data and the second output data.

[0147] In some embodiments, the loss function value includes a first loss function value and a second loss function value.

[0148] It can be understood that after obtaining the first output data, the electronic device can determine the first loss function value of the initial recognition model based on the first loss function between the first output data and a labeled gate image corresponding to the first output data.

[0149] It can be understood that the recognition network model can construct a first loss function between the output and input of the model to determine the loss of the model. Here, the output of the model is the first output data (the heat map matrix of the model output), and the input of the model is the heat map matrix determined based on the labeled gate image. Therefore, by comparing the first output data and a labeled gate image corresponding to the first output data, that is, according to the first loss function between the first output data and a labeled gate image corresponding to the first output data, the loss of the recognition network model can be obtained.

[0150] In some embodiments, in order to adapt to different recognition task requirements, a first loss function corresponding to the recognition task requirements may be constructed for the initial recognition model. In some embodiments, the first loss function may be a cross-entropy loss function (Cross-Entropy Loss).

[0151] It can be understood that when the initial recognition model has a second output layer, the initial recognition model can also determine the second loss function value of the initial recognition model based on the second output data output by the second output layer. That is, by constructing a second loss function between the second output data and the corresponding annotated gate image, the second loss function value of the initial recognition model can be determined.

[0152] It can be understood that the recognition network model can construct a second loss function between the output and input of the model to determine the loss of the model. Here, the output of the model is the second output data, and the input of the model is the position code determined according to the annotated gate image. Therefore, by comparing the second output data with an annotated gate image corresponding to the second output data, that is, according to the second loss function between the second output data and an annotated gate image corresponding to the second output data, the loss of the initial recognition model can be obtained.

[0153] In some embodiments, step S703 may include: dividing the first output data into positive samples and negative samples, and determining a first loss function value of a first loss function according to the positive samples, the negative samples and the labeled gate images.

[0154] It can be understood that the electronic device can divide the first output data into positive samples and negative samples, the positive samples include the first image data of multiple key points recognized by the initial recognition model in the labeled gate image, and the negative samples include the image data other than the first image data in the labeled gate image. The positive sample loss is determined by comparing the positive sample with the labeled gate image, and the negative sample loss is determined by comparing the negative sample with the labeled gate image, and finally the first loss function value of the first loss function is obtained based on the positive sample loss and the negative sample loss.

[0155] Among them, the positive sample can be a set of thermal areas whose thermal values ​​are greater than a preset threshold; the negative sample can be a set of thermal areas whose thermal values ​​are less than the preset threshold.

[0156] It can be understood that the electronic device can divide the first output data into positive samples and negative samples, the positive samples include the first image data of multiple key points recognized by the initial recognition model in the labeled gate image, and the negative samples include the image data other than the first image data in the labeled gate image. The positive sample loss is determined by comparing the positive sample with the labeled gate image, and the negative sample loss is determined by comparing the negative sample with the labeled gate image, and finally the first loss function value of the first loss function is obtained based on the positive sample loss and the negative sample loss.

[0157] Exemplarily, the thermal value of each heat region in the positive sample may be greater than 0.3, and the thermal value of each heat region in the negative sample may be less than or equal to 0.3.

[0158] In some embodiments, step S703 may include: determining distance feature data of each key point in the plurality of key points based on the second output data; and determining a second loss function value of the second loss function according to the distance feature data and the annotated gate image.

[0159] It can be understood that the electronic device can determine the distance feature data of each key point in the multiple key points according to the second output data of the key point. Here, the distance feature data of each key point can characterize the distance feature of each key point from a preset position on the annotated gate image. According to the distance feature data of each key point and the annotated gate image, the first loss and the second loss are determined. Then, according to the first loss and the second loss, the second loss function value of the second loss function is determined.

[0160] In one embodiment, the preset position may be the upper left corner of the marked gate image.

[0161] In some embodiments, the first loss is used to characterize the loss between related key points, and the second loss is used to characterize the loss between unrelated key points. Here, the related key points may be two key points with similar distance feature data, and the unrelated key points may be two key points without similar distance feature data.

[0162] It can be understood that the initial recognition model encourages the embedding vectors of two key points with similar distance feature data to be close to each other through the first loss and the second loss, and encourages the embedding vectors of two key points with dissimilar distance feature data to be far away from each other.

[0163] Step S704: Optimize the model parameters of the initial recognition model based on the loss function value to obtain a recognition network model.

[0164] It can be understood that after obtaining the first loss function value and the second loss function value, the electronic device can adjust the network parameters of the initial recognition model based on the first loss function value and the second loss function value, and repeat the above steps until the loss function value meets the preset requirements to obtain a trained recognition network model.

[0165] In an embodiment of the present application, the first output layer of the initial recognition model can output first output data, and the second output layer can output second output data of key points. Based on the first output data, the second output data of key points, and the labeled gate image input into the initial recognition model, the loss of the initial recognition model can be determined, thereby optimizing the initial recognition model and obtaining a recognition network model.

[0166] Below, a specific example is used to introduce the method of building a recognition network model.

[0167] Step 1: Data preparation: Obtain a manually annotated dataset, which includes one or more annotated gate images.

[0168] For example, when the gate wing of the gate machine is a single pole (hereinafter referred to as the toll pole), the four vertices of the rectangular box circumscribing the toll pole are manually annotated. The annotated data all contain the original natural image (short-focus image) collected and the corresponding toll pole label. The training data set and the test data set are both from the annotated data set, and the image size is 3840*2160 (pixel*pixel). Among them, the training data set contains more than 20,000 images, and the test data set contains more than 1,000 images.

[0169] In one example, the annotated data of the toll pole is interpolated, referring to Figure 8The serial number of the midpoint, insert a point between points 1 and 4, record it as m1; insert a point between points 2 and 3, record it as m2. Calculate the midpoint of points 1 and 2, record it as the top point; calculate the midpoint of points 3 and 4, record it as the down point; calculate the midpoint of m1 and m2, record it as the middle point.

[0170] It is understandable that if the middle point of the toll pole is directly marked, the annotator needs to rely on his senses to determine where the middle point of the toll pole is, and the perception of the middle point position varies from person to person, so there will be a certain amount of marking error. In the embodiment of the present application, the vertices of the toll pole are used as marking points, which can greatly reduce the deviation caused by the sensory differences of the annotators, because the vertices are easier to define and clear, making the marking results more stable.

[0171] Step 2: Build a data iterator: including two parts of data.

[0172] The first part of data:

[0173] In one example, the image is enlarged or reduced to change the image size to 960*512 (pixel*pixel). At least three heat map matrices corresponding to three key position points are constructed, including a heat map matrix corresponding to the top point, a heat map matrix corresponding to the middle point, and a heat map matrix corresponding to the down point.

[0174] In one example, taking the heat map matrix corresponding to the top point as an example, according to the hyperparameter (stride) set by the model, the shape of the heat map matrix can be [Batch, 1, H, W], where Batch represents the channel; H represents the height dimension (height), which is used to represent the height of the image or feature map. W represents the width dimension width. Here, W*stride = 960, H*stride = 512.

[0175] It can be understood that through the above settings, each element position of the two matrices corresponds to a stride*stride image area, which can effectively utilize the position information of the element, and the value of the element can be used to represent other information. This design can greatly reduce the amount of calculation of the model.

[0176] In one example, the data in the heat map matrix ranges from 0 to 1, where 0 means that there is no top point of the toll gate at that location (corresponding to a pixel block on the annotated gate image), and 1 means that there is a top point of the toll gate at that location. At the same time, there are some non-zero values ​​less than 1 near 1, and these values ​​satisfy the Gaussian distribution. The closer the value is to the top point, the closer it is to 1, and the farther the value is from the top point, the closer it is to 0.

[0177] Similarly, we can obtain the heat map matrix corresponding to the middle point and the heat map matrix corresponding to the down point. Fig.10 An optional schematic diagram of the first output data provided in the embodiment of the present application. Fig.10 As shown, according to Fig. 9 The heat map matrix is ​​constructed from the labeled gate image shown in the figure. According to the heat map matrix, the following can be obtained: Fig.10 The first output data shown. Fig.10 (a) in the figure shows the first output data corresponding to the top point. Fig.10 (b) in the figure shows the first output data corresponding to the middle point. Fig.10 (c) in FIG. 1 shows the first output data corresponding to the down point.

[0178] It should be noted that when the recognition network model outputs the first output data, the first output data may be a key point heat map.

[0179] The second part of data:

[0180] In one example, a first matrix and a second matrix are created to represent the distance from the top point to the upper left corner and the distance from the down point to the upper left corner, respectively. Taking the top point as an example, the distance between the top point and the upper left corner of the first output data is calculated (top_y*heatmap_w+top_x). Among them, top_y represents the position of the top point in the y direction, heatmap_w represents the width dimension of the first output data, and top_x represents the position of the top point in the x direction. Similarly, the distance of the down point relative to the upper left corner can be obtained.

[0181] Step 3: Build the network: Use vargnet as the backbone network to obtain the features of the image.

[0182] In one example, the initial recognition model uses vargnet to obtain the features of the image. The output layer uses two output layers, one output layer outputs the first output data, and the other output layer outputs the second output data of the key points. The features output by the two output layers are used to calculate the top point, middle point, down point and the pull loss function and push loss function respectively.

[0183] Understandably, the advantages of vargnet can include: a. Lightweight network: vargnet is a lightweight network suitable for embedded system computing, which reduces on-chip / off-chip memory communication and improves model efficiency. b. Balanced computational intensity: vargnet makes the computational intensity of operations in the network more balanced by fixing the number of channels in each group rather than the number of groups, making it easier to optimize. c. Large network capacity: Compared with lightweight convolutional neural networks (mobilenet), vargnet has a larger network capacity, allows a smaller channel number, and helps to reduce the time consumption of off-chip communication.

[0184] Step 4: Construct loss: It includes two parts of loss function, which correspond to the two parts of data in the data iterator respectively.

[0185] The first part of the loss function:

[0186] In one example, a focal loss function is constructed. The focal loss function is shown in formula (2).

[0187] FL(pt)=-αt(1-pt)γlog(pt) (2)

[0188] Among them, pt represents the model's predicted probability of the sample, αt represents the weight of the sample, and γ is an adjustable parameter that controls the weight difference between difficult and easy samples.

[0189] It is understandable that the purpose of selecting the focalloss function is to give higher weights to samples that are difficult to classify (i.e., samples close to the category boundary) during training, thereby improving the model's classification performance for these samples. By adjusting the parameters in the initial recognition model, such as the beta parameter and the alpha parameter, the weights of positive and negative samples can be controlled to adapt to different data distributions and task requirements.

[0190] In one example, the first part of the loss function includes the focalloss function of the positive sample and the focalloss function of the negative sample, which are specifically used to process the heat map matrix. Specifically, get the positive sample mask (pos_mask) and the negative sample mask (neg_mask), calculate the weight of the negative sample (neg_weights), and calculate the positive sample loss (positive sample loss) and the negative sample loss (neg sample loss) respectively. Then calculate the total number of positive samples and the total loss of positive and negative samples, ensure that the number of positive samples is not zero, use the maximum value of a function, get the maximum value that the function value can take, compare it with 1, and take the larger value. Calculate the final loss, add the positive and negative sample losses and divide by the number of positive samples.

[0191] The second part of the loss function:

[0192] In one example, a pull loss function and a push loss function are constructed. The pull loss function encourages similar embedding vectors to be close to each other, while the push loss function encourages dissimilar embedding vectors to be far away from each other.

[0193] In one example, the pull loss function calculates the mean of each pair of embedding vectors, calculates the squared distance between each embedding vector and the mean, and then normalizes it by the number of valid elements. The normalized distance is multiplied by the mask and then summed to obtain part of the pull loss.

[0194] In one example, the push loss function creates a mask (push_mask) that indicates which embedding vector pairs are valid (i.e., not a comparison between themselves). The distance between the embedding vector pairs is calculated, and the activation (ReLU) function is used to ensure that the distance is at least 1. The distance is subtracted by a value based on the number of valid elements, and then divided by a value based on the number of valid elements to obtain part of the final push loss function. Invalid embedding vector pairs are filtered out by the mask, and then summed to obtain the push loss function.

[0195] Step 5: Train the network: After building the initial recognition model, use the training data generated in step 2 under the open source deep learning framework (Pytorch deep learning framework) to generate the heat map matrix corresponding to the top point, middle point, and down point through the data iterator in step 3, as well as the distance between the corresponding top point and down point and the upper left corner of the first output data, and train the network. Set the training hyperparameters, the batch size of the training is 16, and the input image size is 3840*2160 (pixel*pixel). The model is iterated and trained 200,000 times, and the learning rate is 3e-3.

[0196] Step 5: Model deployment and use.

[0197] The network recognition model after iterative training is deployed on the vehicle. The output of the network recognition model is a key point heat map, which represents the results of the top point, middle point, and down point. After post-processing, the threshold is set to 0.3, and then multiplied by the corresponding step to obtain the corresponding coordinates in the original image, and the top point, middle point, and down point of the toll pole are obtained. Here, the top point, middle point, and down point on the key point heat map can be the key point.

[0198] For example, Fig.11 A first optional schematic diagram of the key points provided for an embodiment of the present application. Fig.12A second optional schematic diagram of the key points provided in the embodiment of the present application. Figure 11 to Figure 12 As shown, when the gate wing of the gate machine is a double-pole, the network recognition model identifies multiple key points on the double-pole (for example, the multiple key points are top points, middle points and down points). Fig.11 For the case where the network recognition model correctly identifies the key points, Fig.12 This is the case where the network recognition model mistakenly identifies the key point. For the above two cases, after obtaining the key point, the gate recognition method described in any one of the first aspects needs to be used to further identify the gate.

[0199] In an embodiment of the present application, the key points on the gate are identified by generating a key point heat map. Each gate is treated as an entity and the starting point, midpoint and end point of the gate are marked. Then the starting point, midpoint and end point of each gate are identified through the heat map matrix of key point detection, so as to achieve the purpose of identification. This method has less computational complexity than the segmentation task, and does not require reclassification of the key point results, so classification errors will not occur. Moreover, the model can better adapt to the irregular characteristics of the gate, thereby improving the detection effect of the gate.

[0200] In a second aspect, an embodiment of the present application provides a gate recognition device. Fig.13 This is an optional schematic diagram of a gate identification device provided in an embodiment of the present application. Fig.13 As shown, the gate recognition device 1300 includes:

[0201] The first processing module 1301 is used to obtain the gate image to be identified acquired by the image acquisition device;

[0202] The second processing module 1302 is used to identify multiple key position points in the gate image to be identified by using a recognition network model to generate multiple key point heat maps, wherein the key point heat map includes multiple heat areas, each heat area has a corresponding heat mark, and the heat mark is used to indicate the probability that the corresponding heat area contains the key position point; the multiple key position points include at least one of the gate wing starting point, the gate wing midpoint and the gate wing end point of the gate;

[0203] The third processing module 1303 is used to obtain key points corresponding to multiple key position points according to multiple key point heat maps, wherein one key point heat map obtains a key point corresponding to one key position point;

[0204] The fourth processing module 1304 is used to identify the gate when the position relationship between the multiple key points meets the preset position relationship.

[0205] In some possible implementations, the fourth processing module 1304 is also used to obtain a first key point from multiple key points; based on the first key point, determine multiple second key points from multiple key points, wherein there is a corresponding relationship between a first key position point corresponding to the first key point and a second key position point corresponding to the second key point; the first key position point and the second key position point are included in the key position point; and identify the gate when the positional relationship between the first key point and the multiple second key points satisfies a preset positional relationship.

[0206] In some possible implementations, the fourth processing module 1304 is also used to determine a third key point and a fourth key point from multiple second key points, the distance between the third key position point corresponding to the third key point and the first key position point is equal to the distance between the fourth key position point corresponding to the fourth key point and the first key position point, and the third key position point and the fourth key position point are located on both sides of the first key position point; the third key position point and the fourth key position point are included in the second key position point; determine the first distance between the first key point and the third key point, and the second distance between the first key point and the fourth key point; and identify the gate when the difference between the first distance and the second distance is less than a preset threshold.

[0207] In some possible implementations, the fourth processing module 1304 is also used to determine the orientation of each second key point among multiple second key points relative to the first key point, wherein each second key point corresponds to an orientation; and identify the gate when the orientation difference between the multiple orientations is less than a preset threshold.

[0208] In some possible implementations, the fourth processing module 1304 is also used to obtain a fifth key point corresponding to the starting point of the gate wing and a sixth key point corresponding to the end point of the gate wing from multiple key points; determine the angle between the first direction formed by the fifth key point and the sixth key point and the horizontal direction; when the degree of the angle is greater than a preset threshold, determine that the gate wing is in an open posture.

[0209] In some possible implementations, the third processing module 1303 is also used to determine the key position point corresponding to each key point heat map based on the output channel of each key point heat map in multiple key point heat maps; determine the thermal value of each heat area in each key point heat map based on the heat mark; and determine the heat area with a thermal value greater than a preset threshold as the key point corresponding to the key position point.

[0210] In some possible implementations, the second processing module 1302 is further configured to train model parameters of an initial recognition model using labeled gate images to obtain a recognition network model.

[0211] In a third aspect, an embodiment of the present application provides an electronic device, comprising at least a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the program, the steps in the gate identification method provided in the above embodiment are implemented.

[0212] Fig.14 An optional structural diagram of an electronic device provided in an embodiment of the present application is shown below in combination with Fig.14 The electronic device 1400 shown is used to illustrate the structure of the electronic device.

[0213] In one example, if Fig.14 As shown, the electronic device 1400 includes: a processor 1401, at least one communication bus 1402, at least one external communication interface 1403 and a memory 1404. The communication bus 1402 is configured to realize the connection and communication between these components. The external communication interface 1403 may include a standard wired interface and a wireless interface.

[0214] The memory 1404 is configured to store instructions and applications executable by the processor 1401, and can also cache data to be processed or processed by the processor 1401 and various modules in the electronic device (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0215] In another example, the electronic device may be a smart terminal such as a car computer, a smart phone, etc. The electronic device is used to execute the steps in the gate recognition method provided in the above embodiment.

[0216] In a fourth aspect, the present application further provides a storage medium, on which a readable storage medium stores executable instructions, wherein when the executable instructions are executed by a processor, the steps in the gate identification method provided in the embodiment of the present application are implemented.

[0217] In a fifth aspect, the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the above-mentioned gate identification method.

[0218] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0219] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0220] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0221] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0222] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0223] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0224] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0225] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the gate identification method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0226] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A gate recognition method, characterized in that: include: Obtaining the gate image to be identified acquired by the image acquisition device; By identifying the network model, multiple key position points in the gate image to be identified are identified to generate multiple key point heat maps, wherein the key point heat map includes multiple heat areas, each heat area has a corresponding heat mark, and the heat mark is used to indicate the probability that the corresponding heat area contains the key position point; the multiple key position points include at least one of the gate wing starting point, gate wing midpoint and gate wing end point of the gate; According to the plurality of key point heat maps, key points corresponding to the plurality of key position points are obtained, wherein a key point corresponding to a key position point is obtained from one key point heat map; When the positional relationship between the multiple key points satisfies the preset positional relationship, the gate is identified.

2. The method according to claim 1, characterized in that When the positional relationship between the multiple key points satisfies the preset positional relationship, identifying the gate includes: Obtaining a first key point from the plurality of key points; Based on the first key point, a plurality of second key points are determined from the plurality of key points, wherein a first key position point corresponding to the first key point and a second key position point corresponding to the second key point have a corresponding relationship; and the first key position point and the second key position point are included in the key position points; When the positional relationship between the first key point and the plurality of second key points satisfies a preset positional relationship, the gate is identified.

3. The method according to claim 2, characterized in that When the first key point corresponds to the midpoint of the gate wing, the gate is identified when the positional relationship between the first key point and the plurality of second key points satisfies a preset positional relationship, including: Determine a third key point and a fourth key point from the plurality of second key points, wherein the distance between a third key position point corresponding to the third key point and the first key position point is equal to the distance between a fourth key position point corresponding to the fourth key point and the first key position point, and the third key position point and the fourth key position point are located on both sides of the first key position point; and the third key position point and the fourth key position point are included in the second key position point; Determine a first distance between the first key point and the third key point, and a second distance between the first key point and the fourth key point; When the difference between the first distance and the second distance is less than a preset threshold, the gate is identified.

4. The method according to claim 2, characterized in that: When the first key point corresponds to the gate wing starting point or gate wing end point, and when the positional relationship between the first key point and the plurality of second key points satisfies a preset positional relationship, identifying the gate includes: Determine the position of each second key point in the plurality of second key points relative to the first key point, wherein each second key point corresponds to one of the positions; When the orientation difference between the multiple orientations is less than a preset threshold, the gate is identified.

5. The method according to claim 1, characterized in that When the plurality of key positions include a gate wing starting point and a gate wing end point of the gate, after identifying the gate, the method further includes: Obtaining a fifth key point corresponding to the gate wing starting point and a sixth key point corresponding to the gate wing end point from the plurality of key points; Determine an angle between a first direction formed by the fifth key point and the sixth key point and a horizontal direction; When the degree of the included angle is greater than a preset threshold, it is determined that the gate wing is in an open posture.

6. The method according to claim 1, characterized in that The step of obtaining key points corresponding to the plurality of key position points according to the plurality of key point heat maps includes: Determine the key position point corresponding to each key point heat map according to an output channel of each key point heat map among the multiple key point heat maps; Determine the thermal value of each of the thermal areas in each of the key point heat maps according to the thermal mark; The heat area whose thermal value is greater than a preset threshold is determined as the key point corresponding to the key position point.

7. The method according to claim 1, characterized in that Before identifying a plurality of key position points in the gate image to be identified by using the identification network model, the method further includes: Obtain an annotated gate image; Inputting the labeled gate image into an initial recognition model to obtain first output data and second output data; Determining a loss function value of the initial recognition model according to the first output data and the second output data; Based on the loss function value, optimizing the model parameters of the initial recognition model to obtain the recognition network model; The first output data is used to indicate the heat tags of the multiple heat regions, and the second output data is used to indicate the position codes of the multiple heat regions.

8. A gate recognition device, characterized in that: include: The first processing module is used to obtain the gate image to be identified acquired by the image acquisition device; The second processing module is used to identify multiple key position points in the gate image to be identified by using a recognition network model to generate multiple key point heat maps, wherein the key point heat map includes multiple heat areas, each heat area has a corresponding heat mark, and the heat mark is used to indicate the probability that the corresponding heat area contains the key position point; the multiple key position points include at least one of the gate wing starting point, gate wing midpoint and gate wing end point of the gate; A third processing module is used to obtain key points corresponding to the multiple key position points according to the multiple key point heat maps, wherein one key point heat map obtains a key point corresponding to one key position point; The fourth processing module is used to identify the gate when the position relationship between the multiple key points meets the preset position relationship.

9. An electronic device, characterized in that: The electronic device includes a memory and one or more processors. A computer program is stored in the memory. When the computer program is executed by the electronic device, the gate recognition method as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores executable instructions, wherein when the executable instructions are executed by the processor, the gate identification method as claimed in any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program or instructions, characterized in that: When the computer program or instruction is executed by a processor, the gate recognition method as claimed in any one of claims 1 to 7 is implemented.

Images