A shared bicycle rollover detection method, device, electronic device and storage medium

Abstract

The present invention discloses a method, device, electronic device and storage medium for detecting the overturning of a shared bicycle. The present invention first obtains an image of a bicycle to be detected, then performs feature extraction on the obtained image to obtain a key point detection feature map, and performs key point recognition based on the key point detection feature map to obtain the coordinates of the key points of the two wheels and the key points of the seat of the bicycle to be detected in the image. Then, the height of the seat can be obtained through the coordinates of the key points of the two wheels and the key points of the seat, and then it is judged whether the bicycle to be detected has overturned according to the height of the seat, that is, if the seat height is less than a preset threshold, it means that the bicycle to be detected has overturned, otherwise it has not overturned; through the above design, the present invention adopts machine vision technology to perform overturning recognition of shared bicycles. Compared with traditional manual patrol inspections, it not only improves efficiency but also reduces manpower and material costs, and is suitable for large-scale application and promotion.

Description

Technical Field

[0001] The present invention belongs to the technical field of bicycle rollover identification, and specifically relates to a shared bicycle rollover detection method, device, electronic equipment and storage medium. Background Art

[0002] Shared bicycles refer to the cooperation between enterprises and the government to provide bicycle sharing services in campuses, subway stations, bus stops, residential areas, commercial areas, public service areas, etc. It is a new form of sharing economy. According to the report, my country's shared bicycle market has gone through three stages of development. The first stage was from 2007 to 2010. The public bicycle model that emerged from abroad began to be introduced into China. It was led by the government and assisted by the city. Most of the bicycles were docked bicycles. The second stage was from 2010 to 2014. Enterprises specializing in the bicycle market began to appear, but public bicycles were still mainly docked bicycles. The third stage was from 2014 to the present. With the rapid development of mobile Internet, Internet shared bicycles led by Meituan, Qingju and Hello came into being. The more convenient dockless Internet shared bicycles began to replace docked bicycles. They are spread across various public places and are widely used by the public for their convenience and low prices, bringing great convenience to people's lives and work.

[0003] However, as dockless bicycles, Internet shared bicycles are more likely to overturn than docked bicycles. Once overturned, it is not only easy to damage the vehicle but also affect the cityscape. There is no effective overturning detection method in the existing technology. It depends entirely on patrol inspections by operators, which is not only inefficient but also wastes a lot of manpower and material resources. Therefore, it is urgent to provide a shared bicycle overturning detection method that is efficient and does not require manual labor. Summary of the Invention

[0004] The purpose of the present invention is to provide a shared bicycle rollover detection method, device, electronic device and storage medium to solve the problems of low efficiency and waste of manpower and material resources in the existing technology of using manual patrol to check for shared bicycle rollovers.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, a shared bicycle rollover detection method is provided, comprising:

[0007] Acquire an image to be detected, wherein the image to be detected includes a bicycle to be detected;

[0008] Inputting the bicycle to be detected into a key point detection model to obtain the coordinates of the key points of the two wheels and the key point of the seat on the bicycle to be detected in the image to be detected, so as to obtain a rollover detection result of the bicycle to be detected based on the coordinates of the key points of the two wheels and the key point of the seat, wherein the key point detection model includes a feature extraction network and a key point detection network, and the feature extraction network includes a dimensionality increase layer and multiple feature extraction modules;

[0009] The dimension-raising layer is used to perform dimension-raising processing on the image to be detected to obtain a dimension-raised image;

[0010] The multiple feature extraction modules are arranged in sequence along the image processing direction in the key point detection model, and the first feature extraction module among the multiple feature extraction modules is used to perform feature extraction processing on the up-dimensional image, wherein, for the i-th feature extraction module and the i+1-th feature extraction module among the multiple feature extraction modules, the i+1-th feature extraction module is used to perform feature extraction processing on the feature image output by the i-th feature extraction module to obtain the i+1-th first feature map, and when i+1=n, the i+1-th first feature map output by the i+1-th feature extraction module is used as the key point detection feature map of the image to be detected, i=1,2,3,...,n, where n is the total number of feature extraction modules;

[0011] The key point detection network is used to perform key point detection on the key point detection feature map to obtain the key point coordinates of the two wheels and the key point coordinates of the seat on the bicycle to be detected.

[0012] Based on the above disclosed content, the present invention first obtains an image of a bicycle to be detected, and then performs feature extraction on the obtained image to obtain a key point detection feature map, and performs key point recognition based on the key point detection feature map to obtain the key point coordinates of the two wheels and the key point coordinates of the seat of the bicycle to be detected in the image. The processing process is: first use the dimensionality increase layer in the feature extraction network in the key point detection model to increase the image dimension, thereby increasing the feature expression ability and obtaining richer features. Then, multiple feature extraction modules can be used to perform feature extraction on the increased dimension image in sequence according to the image processing direction to obtain a first feature image with rich feature information, and after multiple feature extraction modules, the image is processed. After the feature extraction processing of the feature extraction module, the final first feature map can be input into the key point detection network to obtain the coordinates of the two wheel key points and the key point coordinates of the seat; then, the height of the seat can be obtained through the coordinates of the two wheel key points and the key point coordinates of the seat, and then the height of the seat can be used to judge whether the bicycle to be detected has overturned, that is, if the seat height is less than the preset threshold, it means that the bicycle to be detected has overturned, otherwise, it has not overturned; through the above design, the present invention uses machine vision technology to perform overturning identification of shared bicycles. Compared with traditional manual patrol inspections, it not only improves efficiency, but also reduces manpower and material costs, and is suitable for large-scale application and promotion.

[0013] In one possible design, for the i+1th feature extraction module among the multiple feature extraction modules, the i+1th feature extraction module includes a downsampling layer, a feature extraction layer, and an image stitching layer arranged in sequence according to an image processing direction;

[0014] The downsampling layer in the (i+1)th feature extraction module is used to downsample the feature image output by the (i)th feature extraction module to obtain a first sampled image, and the size of the first sampled image output by the downsampling layer in the (i+1)th feature extraction module is half the size of the feature image output by the (i)th feature extraction module;

[0015] The feature extraction layer in the (i+1)th feature extraction module is used to perform convolution processing and copy processing on the first sampled image to obtain a second feature map and a copy image;

[0016] The image stitching layer in the (i+1)th feature extraction module is used to perform image stitching on the second feature map and the copied image to obtain the (i+1)th first feature map, wherein the size of the (i+1)th first feature map is the same as that of the second feature map, and the number of channels is the sum of the number of channels of the second feature map and the number of channels of the copied image.

[0017] Based on the above-mentioned disclosure, the present invention discloses a specific network structure of a feature extraction module, that is, for two adjacent feature extraction modules, the downsampling layer in the latter feature extraction module first downsamples the feature map output by the former feature extraction module to obtain a first sampling image. The function of this network layer is to reduce the dimension of the feature and retain effective information, thereby avoiding overfitting; and the feature extraction layer is used to perform convolution processing on the first sampling image, thereby realizing feature extraction and obtaining a second feature map. At the same time, it is copied to facilitate subsequent channel splicing of the two, that is, the image splicing layer performs channel splicing on the second feature map and the copied image to obtain a first feature map with unchanged size but increased number of channels. In this way, the feature extraction module provided by the present invention can, first, establish a complete connection between all the previous layers and the subsequent layers, second, realize feature reuse by connecting the features on the channel, and third, save calculations. Thus, the richness of the extracted features can be further improved while reducing the computational complexity.

[0018] In one possible design, the feature extraction layer includes an N-channel convolution layer and a copy layer, wherein the N-channel convolution layer is used to perform convolution processing on the first sampling image to obtain a second feature map with N channels, and the copy layer is used to copy the first sampling image to obtain the copied image, and each channel of the N-channel convolution layer uses the same convolution kernel, and N is a positive integer.

[0019] Based on the above disclosure, the convolution layer provided by the present invention is a convolution kernel that performs one operation on each channel, with a computational cost of 1xN; while in traditional convolution operations, one operation is performed on each channel and each convolution kernel. Therefore, when the number of channels is N, if the number of convolution kernels is K, the computational cost is NxK; in this way, the present invention can significantly reduce the computational cost.

[0020] In a possible design, the dimension-raising layer uses a convolutional layer with 32 channels, a convolution kernel size of 3×3, 1 zero padding number, and a stride of 1.

[0021] In one possible design, when there are multiple feature extraction layers, the multiple feature extraction layers are arranged in sequence according to the image processing direction, and the latter feature extraction layer of two adjacent feature extraction layers is used to perform convolution processing and copy processing on the image output by the previous feature extraction layer.

[0022] In one possible design, there are 5 feature extraction modules, among which the first feature extraction module is provided with 2 feature extraction layers, the second feature extraction module and the third feature extraction module are each provided with 3 feature extraction layers, the fourth feature extraction module is provided with 4 feature extraction layers, and the fifth feature extraction module is provided with 1 feature extraction layer.

[0023] In one possible design, a grid conversion network is further provided between the feature extraction network and the key point detection network, wherein the grid conversion network is used to convert the key point detection feature map into a grid image of a preset size, and the key point detection network is used to perform key point detection on the grid image to obtain the key point coordinates of the two wheels and the key point coordinates of the seat on the bicycle to be detected; through the above design, the grid conversion network can convert the first feature map into a grid of a fixed size, thereby ensuring the accuracy of the spatial position during subsequent key point recognition.

[0024] In a second aspect, a shared bicycle rollover detection device is provided, comprising:

[0025] An image acquisition unit, configured to acquire an image to be detected, wherein the image to be detected includes a bicycle to be detected;

[0026] a tipping detection unit, configured to input the bicycle to be detected into a key point detection model, and obtain the coordinates of the key points of the two wheels and the key point of the seat on the bicycle to be detected in the image to be detected, so as to obtain a tipping detection result of the bicycle to be detected based on the coordinates of the key points of the two wheels and the key point of the seat, wherein the key point detection model includes a feature extraction network and a key point detection network, and the feature extraction network includes a dimensionality increase layer and a plurality of feature extraction modules;

[0027] The dimension-raising layer is used to perform dimension-raising processing on the image to be detected to obtain a dimension-raised image;

[0028] The multiple feature extraction modules are arranged in sequence along the image processing direction in the key point detection model, and the first feature extraction module among the multiple feature extraction modules is used to perform feature extraction processing on the up-dimensional image, wherein, for the i-th feature extraction module and the i+1-th feature extraction module among the multiple feature extraction modules, the i+1-th feature extraction module is used to perform feature extraction processing on the feature image output by the i-th feature extraction module to obtain the i+1-th first feature map, and when i+1=n, the i+1-th first feature map output by the i+1-th feature extraction module is used as the key point detection feature map of the image to be detected, i=1,2,3,...,n, where n is the total number of feature extraction modules;

[0029] The key point detection network is used to perform key point detection on the key point detection feature map to obtain the key point coordinates of the two wheels and the key point coordinates of the seat on the bicycle to be detected.

[0030] In the third aspect, a shared bicycle rollover detection device is provided. Taking the device as an electronic device as an example, it includes a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the shared bicycle rollover detection method as described in the first aspect or any possible design of the first aspect.

[0031] In a fourth aspect, a storage medium is provided, on which instructions are stored. When the instructions are run on a computer, the shared bicycle rollover detection method as described in the first aspect or any possible design of the first aspect is executed.

[0032] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, causes the computer to execute the shared bicycle rollover detection method as described in the first aspect or any possible design of the first aspect.

[0033] Beneficial effects:

[0034] (1) The present invention uses machine vision technology to identify the overturned shared bicycles. Compared with traditional manual patrol inspections, it not only improves efficiency but also reduces manpower and material costs, making it suitable for large-scale application and promotion.

[0035] (2) The feature extraction module provided by the present invention can, firstly, establish a complete connection between all previous layers and subsequent layers, secondly, realize feature reuse by connecting features on channels, and thirdly, save calculations, thereby further improving the richness of extracted features while reducing computational complexity.

[0036] (3) The convolution layer provided by the present invention is a convolution kernel that performs one operation on each channel, and the computational cost is 1xN. In contrast, in traditional convolution operations, one operation is performed on each channel and each convolution kernel. Therefore, when the number of channels is N, if the number of convolution kernels is K, the computational cost is NxK. In this way, the present invention can significantly reduce the computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic flow chart of the steps of a shared bicycle rollover detection method provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the architecture of a key point detection model provided by an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of the structure of a feature extraction network provided by an embodiment of the present invention;

[0040] Figure 4A schematic diagram of the structure of an N-channel convolutional layer provided by an embodiment of the present invention;

[0041] Figure 5 Schematic diagram of the effect of the shared bicycle model provided by an embodiment of the present invention;

[0042] Figure 6 A schematic diagram of the structure of a shared bicycle rollover detection device provided by an embodiment of the present invention;

[0043] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be briefly introduced below in conjunction with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0045] It should be understood that although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element without departing from the scope of the exemplary embodiments of the present invention.

[0046] It should be understood that the term "and / or" that may appear in this document is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may indicate three situations: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" that may appear in this document describes another type of association object relationship, indicating that two relationships may exist. For example, A / and B may indicate two situations: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0047] Example:

[0048] See also Figure 1As shown, the shared bicycle overturning detection method provided in this embodiment adopts machine vision technology to perform overturning identification of shared bicycles, that is, by performing feature recognition on the shared bicycle image to extract the coordinates of the wheel key points and the seat key points of the bicycle to be detected in the image, so that the seat height of the bicycle to be detected can be obtained based on the aforementioned key point coordinates. Finally, based on the seat height, it can be judged whether the bicycle to be detected has overturned. Therefore, compared with traditional manual patrol inspections, the present invention does not require manual labor and has higher detection efficiency; in this embodiment, the method can be, but is not limited to, run on the server side or the AI (Artificial Intelligence) camera side. It can be understood that the aforementioned execution subject does not constitute a limitation on the embodiments of the present application. Accordingly, the operation steps of this method can be, but are not limited to, as shown in the following steps S1 to S2.

[0049] S1. Obtain an image to be detected, wherein the image to be detected includes a bicycle to be detected. In specific applications, it can be, but is not limited to, arranging AI cameras in the parking areas of each shared bicycle, and using the AI cameras to collect images of the shared bicycle parking areas to obtain images to be detected. Of course, it is also possible to collect videos of the parking areas and process the videos frame by frame to obtain multiple images to be detected. Furthermore, it is also possible to use cameras installed on the target street to capture images of the parking areas as images to be detected, and upload them to the server to achieve rollover detection of the bicycle to be detected in the images to be detected. It is understandable that the image acquisition method can be specifically set according to actual use and is not limited to the aforementioned examples.

[0050] After obtaining the image to be detected, it can be input into the key point detection model for feature extraction to obtain a key point feature map, and the key point coordinates of the two wheels and the key point coordinates of the seat on the bicycle to be detected in the image to be detected can be obtained based on the key point feature map. Then, the overturning detection can be performed based on the key point coordinates of the two wheels and the key point coordinates of the seat, wherein the overturning detection step is shown in the following step S2.

[0051] S2. Input the bicycle to be detected into the key point detection model to obtain the coordinates of the two wheel key points and the seat key point on the bicycle to be detected in the image to be detected, so as to obtain the seat height of the bicycle to be detected based on the two wheel key point coordinates and the seat key point coordinates, and obtain the overturning detection result of the bicycle to be detected according to the seat height; in specific application, the coordinates of the two wheel key points and the seat key point coordinates are both two-dimensional coordinates. Therefore, this embodiment needs to first obtain the three-dimensional mapping parameters by performing three-dimensional mapping on the coordinates of the two wheel key points, and then, based on the three-dimensional mapping parameters, the z-axis coordinate value in the three-dimensional coordinate of the seat key point can be obtained. Finally, the z-axis coordinate value in the three-dimensional coordinate of the seat key point can be used as the seat height. When the seat height is higher than a preset threshold, it means that the bicycle to be detected has not overturned. Otherwise, it means that the bicycle to be detected has overturned. Optionally, the solution process of the aforementioned three-dimensional mapping parameters will be explained later.

[0052] See also Figure 2 and Figure 3 As shown, the following discloses one specific architecture of the key point detection model in this embodiment:

[0053] In this embodiment, the key point detection model may include, but is not limited to: a feature extraction network and a key point detection network, and the feature extraction network includes a dimensionality increase layer and a plurality of feature extraction modules; in specific applications, the dimensionality increase layer is used to perform dimensionality increase processing on the image to be detected to obtain an increased dimensionality image. In this way, the dimensionality increase of the image to be detected can increase the feature expression ability and obtain richer features; optionally, for example, the image to be detected is an RGB image, and this embodiment can use a convolution layer with 32 channels, a convolution kernel size of 3×3, a zero padding number of 1 and a step size of 1 to perform dimensionality increase on the image to be detected to obtain an increased dimensionality image with a size of 416*416; in specific implementation, the plurality of feature extraction modules are arranged in sequence along the image processing direction in the key point detection model, wherein the processing process of the increased dimensionality image by the plurality of feature extraction modules is: the first feature extraction module among the plurality of feature extraction modules is used to perform feature extraction processing on the increased dimensionality image, wherein, for the i-th feature extraction module and the i+1-th feature extraction module among the plurality of feature extraction modules, Feature extraction module, the i+1th feature extraction module is used to perform feature extraction processing on the feature image output by the i-th feature extraction module to obtain the i+1th first feature map, and when i+1=n, the i+1th first feature map output by the i+1th feature extraction module is used as the key point detection feature map of the image to be detected, i=1,2,3,...,n, and n is the total number of feature extraction modules; for example, the first feature extraction module performs feature extraction processing on the up-dimensional image to obtain the first first feature map, and the second feature extraction module performs feature extraction processing on the first first feature map to obtain the second first feature map. Similarly, the third feature extraction module performs feature extraction processing on the second first feature map to obtain the third first feature map. Based on this principle, the feature map output by the previous feature extraction module is continuously extracted using the subsequent feature extraction module until the last feature extraction module is polled, and the first feature map output by the last feature extraction module is used as the feature map for key point detection (i.e., the key point detection feature map).

[0054] After using the feature extraction network to extract the key point detection feature map of the image to be detected, the key point detection feature map can be input into the key point detection network for key point detection, that is: the key point detection network is used to perform key point detection on the key point detection feature map to obtain the key point coordinates of the two wheels and the key point coordinates of the seat on the bicycle to be detected.

[0055] Optionally, in this embodiment, for example, a grid conversion network is further provided between the feature extraction network and the key point detection network. The grid conversion network is used to convert the key point detection feature map into a grid image of a preset size. Correspondingly, the key point detection network is used to perform key point detection on the grid image to obtain the key point coordinates of the two wheels and the key point coordinates of the seat on the bicycle to be detected; in this way, converting the first feature map into a grid of a fixed size can ensure the accuracy of the spatial position during subsequent key point recognition, thereby improving the accuracy of key point recognition.

[0056] Furthermore, the grid conversion network can be, but is not limited to, a RoIAlign network (which is a regional feature aggregation network in the RCNN network. The principle is: cancel the quantization operation and use the bilinear interpolation method to obtain the image value at the pixel point with floating-point coordinates, thereby converting the entire feature aggregation process into a continuous operation to obtain a fixed-size grid image based on bilinear interpolation). Of course, the size of the grid image can be set according to the actual use. In addition, the key point detection network can be, but is not limited to, a MonoGRNet V2 model. The overall algorithm framework of the MonoGRNet V2 model is similar to the Mask RCNN framework (which consists of a faster rcnn and a semantic segmentation algorithm FCN. The former completes the target detection task, and the latter can accurately complete the semantic segmentation task). However, the MonoGRNet V2 model adds two additional head branches, one of which is used to regress 2D points, visibility, and its own rotation angle; the other branch is responsible for selecting the CAD model and regressing the 3D size deviation. Therefore, this embodiment uses the MonoGRNet V2 model to first detect key points in the 2D image (i.e., the image to be detected), and then uses the MonoGRNet The V2 model performs 3D regression of two-dimensional key points (i.e., the two wheel key points and the seat key points). Of course, the 3D regression process of two-dimensional coordinates is described in detail below.

[0057] In this way, the key point detection model provided in this embodiment can be used to accurately identify the key point coordinates of the two wheels and the key point coordinates of the seat of the bicycle to be detected, thereby achieving accurate rollover recognition of the bicycle to be detected.

[0058] See also Figure 3 As shown, one of the specific structures of the following feature extraction modules:

[0059] In this embodiment, since the main structure of each feature extraction module is the same, the only difference is the number of layers of the internal network layer. Therefore, the following is a specific explanation using the (i+1)th feature extraction module as an example:

[0060] Optionally, for example, the i+1th feature extraction module includes a downsampling layer, a feature extraction layer and an image stitching layer arranged in sequence according to the image processing direction, wherein the downsampling layer in the i+1th feature extraction module is used to downsample the feature image output by the i-th feature extraction module to obtain a first sampling image; in this embodiment, the function of downsampling is to reduce the dimension of the feature and retain valid information, thereby avoiding overfitting; in addition, the size of the first sampling image output by the downsampling layer in the i+1th feature extraction module is half the size of the feature image output by the i-th feature extraction module, so that when the latter feature extraction module downsamples the feature map output by the previous feature extraction module, the size of the first sampling image obtained is half of the feature map output by the previous feature extraction module; and after passing through the downsampling layer, it can be input into the feature extraction layer for feature re-extraction, that is, the i+1th feature extraction module. The feature extraction layer in the feature extraction module is used to perform convolution processing and copy processing on the first sampling image respectively to obtain a second feature map and a copy image; in specific implementation, the feature extraction layer can realize continuous feature extraction by continuously convolution processing on the first sampling image, thereby obtaining a feature map that can best characterize the bicycle to be detected, and the copy processing is used to provide a data basis for the subsequent channel number splicing; finally, the image splicing layer in the i+1th feature extraction module is used to perform image splicing on the second feature map and the copy image to obtain the i+1th first feature map. In this embodiment, the size of the i+1th first feature map is the same as that of the second feature map, and the number of channels is the sum of the number of channels of the second feature map and the number of channels of the copy image; thereby, the number of channels of the feature map can be increased while ensuring the size, thereby further increasing the extracted features to ensure the accuracy of subsequent key point recognition.

[0061] For further information, see Figure 3 As shown, for example, the feature extraction layer may include, but is not limited to: an N-channel convolution layer and a copy layer, wherein the N-channel convolution layer is used to perform convolution processing on the first sample image to obtain a second feature map with N channels, and the copy layer is used to copy the first sample image to obtain the copy image, and each channel of the N-channel convolution layer uses the same convolution kernel, and N is a positive integer; at the same time, see Figure 4 As shown, Figure 4 The convolution layer in the CNN has N channels in total, and when N channels are convolved, they all use the same convolution kernel (i.e. Figure 4Conv (convolution), f*f, f is the size). Thus, the convolution operation of this embodiment uses one convolution kernel to perform one operation on each channel, and the computational cost is 1xN. In the traditional convolution operation, each channel and each convolution kernel will be operated once, and the computational cost becomes NxK, where k is the number of convolution kernels. Therefore, when performing the convolution operation in this embodiment, the computational cost can be greatly reduced.

[0062] In addition, in this embodiment, as mentioned above, the main structures of the feature extraction modules are the same, and the only difference is the number of feature extraction layers contained therein and the size of the convolution kernels used. Therefore, when there are multiple feature extraction layers, the multiple feature extraction layers are also arranged in sequence according to the image processing direction, and the latter feature extraction layer of the two adjacent feature extraction layers is used to perform convolution processing and copy processing on the image output by the previous feature extraction layer respectively; for example, assuming that the first feature extraction layer is provided with two feature extraction layers, then the first feature extraction layer A performs convolution processing and copy processing on the first down-sampled image, and then the two images are spliced through the image stitching layer to obtain the first first feature map A, then the first first feature map A enters the second feature extraction layer B for convolution and copy processing, and finally, the image is stitched through the image stitching layer to obtain the first first feature map.

[0063] Similarly, when the first feature extraction layer is set with 3 feature extraction layers, the first feature extraction layer A performs convolution and copy processing on the first down-sampled image, and then splices the two images through the image stitching layer to obtain the first first feature map A. Then, the first first feature map A enters the second feature extraction layer B for convolution and copy processing, and similarly passes through the image stitching layer for image stitching to obtain the first first feature map B. Then, the first first feature map C enters the third feature extraction layer C for convolution and copy processing, and similarly passes through the image stitching layer for image stitching to obtain the first first feature map. Of course, the processing process of different numbers of feature extraction layers is consistent with the above example, and will not be repeated here.

[0064] Preferably, in this embodiment, the feature extraction modules are provided with 5 as an example, see Figure 3 As shown, the first feature extraction module is provided with two feature extraction layers (i.e., 2× indicates that there are two feature extraction layers), the second feature extraction module and the third feature extraction module are each provided with three feature extraction layers, the fourth feature extraction module is provided with four feature extraction layers, and the fifth feature extraction module is provided with one feature extraction layer; therefore, the working process of the feature extraction network provided in this embodiment is:

[0065] See also Figure 3As shown in the figure, first, the image to be detected passes through the dimensionality raising layer, that is, the convolution layer with 32 channels, 3×3 convolution kernel size, 1 zero padding number and step size of 1 is used to raise the dimension of the image to be detected, and a raised dimensional image with a size of 416*416 is obtained; then, it enters the first feature extraction module, that is, the downsampling layer in the first feature extraction module (called the first sampling layer, and the convolution layer with 64 channels, 3×3 convolution kernel size, 1 zero padding number and step size of 2 is used for sampling) is used to obtain the first down-sampled image 1 with a size of 208*208. Then, the copy layer is used to copy the first down-sampled image 1 to obtain the copy image 1, and then the 64-channel convolution layer (the convolution kernel size is 5×5, the zero padding number is 2 and the step size is 1) is used to perform a convolution operation on the first down-sampled image 1 to obtain the second feature. Figure 1 Then, the image stitching layer is used to stitch the copy image 1 and the first downsampled image 1 to obtain the first feature Figure 1 (The size of the image is still 208*208, but the number of channels becomes 64+64=128). At this time, since there are two feature extraction layers in the first feature extraction module, the first feature Figure 1 As the input of the next feature extraction layer, convolution and copy operations are performed again (the convolution kernel size, step size and padding number used are the same as the first time). Finally, the two images obtained by the convolution and copy operations are spliced to obtain the first feature map.

[0066] Similarly, the first feature map is input into the second feature extraction module, and is downsampled by the downsampling layer of the second feature extraction module (called the second downsampling layer, and the convolution layer with a channel number of 128, a convolution kernel size of 3×3, a zero padding number of 1 and a step size of 2) to obtain a first downsampled image 2 with a size of 104*104. Then, the copy layer is used to copy the first downsampled image 2 to obtain a copy image 2, and then the 128-channel convolution layer (the convolution kernel size is 3×3, the zero padding number is 1 and the step size is 1) is used to perform a convolution operation on the first downsampled image 2 to obtain the second feature. Figure 2 Then, the image stitching layer is used to stitch the copy image 2 and the first downsampled image 2 to obtain the second first feature Figure 1(The size of the image is still 104*104, but the number of channels becomes 128+128=256). At this time, since there are three feature extraction layers in the second feature extraction module, two convolutions, copies and splicing processes are performed in sequence according to the same convolution kernel size, step size and padding number. After the processing is completed, the second first feature map can be obtained; similarly, according to the same principle above, the third feature extraction module, the fourth feature extraction module and the fifth feature extraction module are used in sequence to perform feature extraction processing to obtain the key point detection feature map.

[0067] Therefore, through the above detailed explanation of the feature extraction module, the feature extraction module provided in this embodiment can, firstly, establish a complete connection between all the previous layers and the subsequent layers; secondly, it can realize feature reuse through the connection of features on the channel; and thirdly, it can save calculations, thereby further improving the richness of the extracted features while reducing the computational complexity.

[0068] In a possible design, the second aspect of this embodiment provides a 3D regression process of the two-dimensional coordinates of key points based on the first aspect of the embodiment, that is, a process for solving three-dimensional mapping parameters, which may include but is not limited to the following steps S3 to S8.

[0069] In specific applications, the branch used to regress 3D size deviations in the MonoGRNet V2 model is used to perform 3D conversion of the two wheel key points and the seat key points. The specific process is: the MonoGRNet V2 model is used to obtain the bicycle model corresponding to the bicycle to be detected. The bicycle model is marked with several calibration key points that have the same positions as the wheel key points and the seat key points. Therefore, the three-dimensional conversion parameters between the two-dimensional key points and the three-dimensional coordinates can be obtained by combining the calibration key points and the coordinates of the wheel and seat key points. The conversion process is as follows:

[0070] S3. Based on the coordinates of the key points of the seat and the two wheels, the length, the first height and the second height of the bicycle to be tested are obtained, wherein the first height is the distance from the key point of the front wheel to the key point of the seat, the second height is the distance from the key point of the rear wheel to the key point of the seat, and the length is the distance between the key point of the front wheel and the key point of the rear wheel; in this way, after obtaining the aforementioned coordinates of the key points of the seat and the two wheels, the length and the two heights can be calculated based on the coordinates.

[0071] S4. Input the length, first height and second height of the bicycle to be detected into the trained vehicle matching model to obtain the three-dimensional coordinates of multiple calibration key points on the bicycle model corresponding to the bicycle to be detected, wherein the position represented by each calibration key point corresponds one-to-one to the position represented by the target key point, and the target key point includes the coordinates of the seat key point and the coordinates of the two wheel key points; in this embodiment, the trained vehicle matching model is the MonoGRNetV2 model, wherein the MonoGRNet V2 model is provided with a model database, in which multiple 3D CAD bicycle sample models (which may include shared bicycles, mountain bikes, road bike models, etc.) are pre-defined, and the three-dimensional coordinates of the three key points of the wheel and seat are marked for each bicycle sample model as calibration key points (see Figure 5 As shown, Figure 5 In the figure, each key point of calibration is marked. Figure 5 G point, V point and D point in the figure), and the length, first height and second height are marked for the bicycle prototype model; at the same time, the length, first height and second height of multiple model bicycles need to be obtained as the target length, target first height and target second height, so that the target length, first height and second height of multiple model bicycles and the length, first height and second height of each of the aforementioned bicycle prototype models are used as input, and the bicycle prototype model corresponding to each model bicycle is used as output to train the MonoGRNet V2 model; further, when the MonoGRNet V2 model performs bicycle matching, its loss function is the difference between the output value (i.e., the length, first and second heights of the bicycle prototype model corresponding to the model bicycle) and the true value (i.e., the target length, first height and second height of the model bicycle), and when the difference is less than the preset value, the training of the model can be terminated.

[0072] Therefore, based on the branch used to regress 3D size deviation in the aforementioned MonoGRNet V2 model, the corresponding bicycle model can be matched based on the length, first height and second height of the bicycle to be detected, thereby obtaining the three-dimensional coordinates of the calibration key points on the corresponding bicycle model.

[0073] After obtaining the bicycle model corresponding to the bicycle to be detected, the three-dimensional conversion parameters can be calculated based on the three-dimensional coordinates of the calibrated key points in the bicycle model and the two-dimensional coordinates of the key points of the wheels and seat, as shown in the following steps S5 to S6.

[0074] S5. Select any two key points from the seat key points and the two wheel key points as the key point matching group, and screen out the calibration key points with the same representation position as the two key points in the key point matching group from each calibration key point, as the matching key points corresponding to the key point matching group; in specific application, if the front wheel key point and the seat key point are selected to form a key point matching group, then the corresponding calibration key points are: the front wheel key point and the seat key point in the bicycle model corresponding to the bicycle to be tested. Of course, the principle of determining the corresponding calibration key points of the remaining key points is the same as the above example, and will not be repeated here.

[0075] After obtaining the matching key points corresponding to the key point matching group, the three-dimensional conversion parameters of the key point matching group can be calculated based on the two-dimensional coordinates of the two key points in the key point matching group and the three-dimensional coordinates of the two corresponding matching key points, as shown in the following step S6.

[0076] S6. Based on the two-dimensional coordinates of the two key points in the key point matching group and the three-dimensional coordinates of the matching key points corresponding to the key point matching group, the three-dimensional transformation parameters of the key point matching group are calculated. In specific applications, the following formula (1) can be used to calculate the three-dimensional transformation parameters.

[0077]

[0078] In the above formula (1), Z represents the three-dimensional transformation parameter of the key point matching group, f represents the focal length of the camera corresponding to the image to be detected, and r cad is the ratio between the ordinates of the two matching key points corresponding to the key point matching group, h p The absolute value of the difference between the vertical coordinates of two key points in the key point matching group (such as the absolute value of the difference between the vertical coordinates of the front wheel key point and the vertical coordinates of the seat key point mentioned above), h represents the height of the bicycle to be detected (the average of the sum of the first height and the second height).

[0079] After the three-dimensional transformation parameters are obtained, the three-dimensional coordinates of the key points of the seat can be calculated using the three-dimensional transformation parameters, as shown in the following step S7.

[0080] S7. Use the value of the three-dimensional transformation parameter of the key point matching group as the z-axis coordinate value in the three-dimensional coordinates of the seat key point.

[0081] Therefore, through the above design, the z-axis coordinate value of the key three-dimensional coordinates of the seat of the bicycle to be tested can be obtained, and the z-axis coordinate value is used as the height of the bicycle to be tested. When the z-axis coordinate value is greater than the preset threshold, it can be determined that the bicycle to be tested has not overturned, otherwise, it means that the bicycle to be tested has overturned; of course, the preset threshold can be preset. For example, if the seat of a normal bicycle is between 1.45-1.9m, then the threshold is set to 1.45-1.9m. At this time, if it is within the threshold range, it means that the bicycle to be tested has not overturned, and if it is lower than 1.45, it means it has overturned.

[0082] Therefore, through the above detailed explanation of the shared bicycle overturning detection method provided in this embodiment, the present invention first obtains an image of the bicycle to be detected, and then performs feature extraction on the obtained image to obtain a key point detection feature map, and performs key point recognition based on the key point detection feature map to obtain the key point coordinates of the two wheels and the key point coordinates of the seat of the bicycle to be detected in the image. Then, the height of the seat can be obtained by the key point coordinates of the two wheels and the key point coordinates of the seat, and then the height of the seat can be used to determine whether the bicycle to be detected has overturned, that is, if the seat height is less than the preset threshold, it means that the bicycle to be detected has overturned, otherwise, it has not overturned; through the above design, the present invention uses machine vision technology to perform overturning recognition of shared bicycles. Compared with traditional manual patrol inspections, it not only improves efficiency, but also reduces manpower and material costs, and is suitable for large-scale application and promotion.

[0083] like Figure 6 As shown, the third aspect of this embodiment provides a hardware device for implementing the shared bicycle rollover detection method described in the first and second aspects of the embodiment, including:

[0084] An image acquisition unit, configured to acquire an image to be detected, wherein the image to be detected includes a bicycle to be detected;

[0085] The tipping detection unit is used to input the bicycle to be detected into the key point detection model, and obtain the coordinates of the key points of the two wheels and the key points of the seat on the bicycle to be detected in the image to be detected, so as to obtain the seat height of the bicycle to be detected based on the coordinates of the key points of the two wheels and the key points of the seat, so as to obtain the tipping detection result of the bicycle to be detected according to the seat height, wherein the key point detection model includes a feature extraction network and a key point detection network, and the feature extraction network includes a dimensionality increase layer and multiple feature extraction modules.

[0086] The dimension-increasing layer is used to perform dimension-increasing processing on the image to be detected to obtain a dimension-increasing image.

[0087] The multiple feature extraction modules are arranged in sequence along the image processing direction in the key point detection model, and the first feature extraction module among the multiple feature extraction modules is used to perform feature extraction processing on the up-dimensional image, wherein, for the i-th feature extraction module and the i+1-th feature extraction module among the multiple feature extraction modules, the i+1-th feature extraction module is used to perform feature extraction processing on the feature image output by the i-th feature extraction module to obtain the i+1-th first feature map, and when i+1=n, the i+1-th first feature map output by the i+1-th feature extraction module is used as the key point detection feature map of the image to be detected, i=1,2,3,...,n, and n is the total number of feature extraction modules.

[0088] The key point detection network is used to perform key point detection on the key point detection feature map to obtain the key point coordinates of the two wheels and the key point coordinates of the seat on the bicycle to be detected.

[0089] The working process, working details and technical effects of the device provided in this embodiment can be found in the first and second aspects of the embodiment, and will not be described in detail here.

[0090] like Figure 7 As shown, the fourth aspect of this embodiment provides another shared bicycle rollover detection device, taking the device as an electronic device as an example, comprising: a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the shared bicycle rollover detection method as described in the first aspect and / or the second aspect of the embodiment.

[0091] For example, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in first-out memory (FIFO), and / or first-in last-out memory (FILO); specifically, the processor may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor may be implemented in at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Furthermore, the processor may include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); and the coprocessor is a low-power processor for processing data in a standby state.

[0092] In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. For example, the processor may be, but is not limited to, a microprocessor of the STM32F105 series, a reduced instruction set computer (RISC) microprocessor, an X86 architecture processor, or a processor with an integrated embedded neural network processing unit (NPU); the transceiver may be, but is not limited to, a wireless fidelity (WIFI) wireless transceiver, a Bluetooth wireless transceiver, a general packet radio service technology (GPRS) wireless transceiver, a ZigBee protocol (a low-power local area network protocol based on the IEEE802.15.4 standard, ZigBee) wireless transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. In addition, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0093] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first and second aspects of the embodiment, and will not be repeated here.

[0094] The fifth aspect of this embodiment provides a storage medium that stores instructions for the shared bicycle overturn detection method described in the first aspect of the embodiment, that is, the storage medium stores instructions, and when the instructions are run on a computer, the shared bicycle overturn detection method described in the first aspect and / or the second aspect is executed.

[0095] The storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive and / or a memory stick, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0096] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first and second aspects of the embodiment, and will not be repeated here.

[0097] A sixth aspect of this embodiment provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the shared bicycle overturning detection method as described in the first and / or second aspects of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0098] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A shared bicycle rollover detection method, characterized in that: include: Acquire an image to be detected, wherein the image to be detected includes a bicycle to be detected; Inputting the bicycle to be detected into a key point detection model to obtain the coordinates of the key points of the two wheels and the key point of the seat of the bicycle to be detected in the image to be detected, so as to obtain the seat height of the bicycle to be detected based on the coordinates of the key points of the two wheels and the key point of the seat, and to obtain a rollover detection result of the bicycle to be detected according to the seat height, wherein the key point detection model includes a feature extraction network and a key point detection network, and the feature extraction network includes a dimensionality increase layer and multiple feature extraction modules; The dimension-raising layer is used to perform dimension-raising processing on the image to be detected to obtain a dimension-raised image; The multiple feature extraction modules are arranged in sequence along the image processing direction in the key point detection model, and the first feature extraction module among the multiple feature extraction modules is used to perform feature extraction processing on the up-dimensional image, wherein, for the i-th feature extraction module and the i+1-th feature extraction module among the multiple feature extraction modules, the i+1-th feature extraction module is used to perform feature extraction processing on the feature image output by the i-th feature extraction module to obtain the i+1-th first feature map, and when i+1=n, the i+1-th first feature map output by the i+1-th feature extraction module is used as the key point detection feature map of the image to be detected, i=1,2,3,...,n, where n is the total number of feature extraction modules; For the (i+1)th feature extraction module among the multiple feature extraction modules, wherein the (i+1)th feature extraction module includes a downsampling layer, a feature extraction layer, and an image stitching layer arranged in sequence according to an image processing direction; The downsampling layer in the (i+1)th feature extraction module is used to downsample the feature image output by the (i)th feature extraction module to obtain a first sampled image, and the size of the first sampled image output by the downsampling layer in the (i+1)th feature extraction module is half the size of the feature image output by the (i)th feature extraction module; The feature extraction layer in the (i+1)th feature extraction module is used to perform convolution processing and copy processing on the first sampled image to obtain a second feature map and a copy image; The image stitching layer in the (i+1)th feature extraction module is used to stitch the second feature map and the copied image to obtain an (i+1)th first feature map, wherein the (i+1)th first feature map has the same size as the second feature map and has a number of channels that is the sum of the number of channels of the second feature map and the number of channels of the copied image; The key point detection network is used to perform key point detection on the key point detection feature map to obtain the key point coordinates of the two wheels and the key point coordinates of the seat on the bicycle to be detected.

2. The method according to claim 1, characterized in that The feature extraction layer includes an N-channel convolution layer and a copy layer, wherein the N-channel convolution layer is used to perform convolution processing on the first sampling image to obtain a second feature map with N channels, and the copy layer is used to copy the first sampling image to obtain the copied image, and each channel of the N-channel convolution layer uses the same convolution kernel, and N is a positive integer.

3. The method according to claim 1, characterized in that The dimension-raising layer uses a channel number of 32 and a convolution kernel size of , convolutional layers with zero padding of 1 and stride of 1.

4. The method according to claim 1, wherein When there are multiple feature extraction layers, the multiple feature extraction layers are arranged in sequence according to the image processing direction, and the latter feature extraction layer of two adjacent feature extraction layers is used to perform convolution processing and copy processing on the image output by the previous feature extraction layer.

5. The method according to claim 4, characterized in that There are five feature extraction modules, among which the first feature extraction module is provided with two feature extraction layers, the second feature extraction module and the third feature extraction module are provided with three feature extraction layers, the fourth feature extraction module is provided with four feature extraction layers, and the fifth feature extraction module is provided with one feature extraction layer.

6. The method according to claim 1, characterized in that A grid conversion network is also provided between the feature extraction network and the key point detection network, wherein the grid conversion network is used to convert the key point detection feature map into a grid image of a preset size, and the key point detection network is used to perform key point detection on the grid image to obtain the key point coordinates of the two wheels and the key point coordinates of the seat on the bicycle to be detected.

7. A shared bicycle rollover detection device, characterized in that: include: An image acquisition unit, configured to acquire an image to be detected, wherein the image to be detected includes a bicycle to be detected; a tipping detection unit, configured to input the bicycle to be detected into a key point detection model, and obtain the coordinates of the key points of the two wheels and the key point of the seat of the bicycle to be detected in the image to be detected, so as to obtain the seat height of the bicycle to be detected based on the coordinates of the key points of the two wheels and the key point of the seat, and to obtain a tipping detection result of the bicycle to be detected based on the seat height, wherein the key point detection model includes a feature extraction network and a key point detection network, and the feature extraction network includes a dimensionality increase layer and a plurality of feature extraction modules; The dimension-raising layer is used to perform dimension-raising processing on the image to be detected to obtain a dimension-raised image; The multiple feature extraction modules are arranged in sequence along the image processing direction in the key point detection model, and the first feature extraction module among the multiple feature extraction modules is used to perform feature extraction processing on the up-dimensional image, wherein, for the i-th feature extraction module and the i+1-th feature extraction module among the multiple feature extraction modules, the i+1-th feature extraction module is used to perform feature extraction processing on the feature image output by the i-th feature extraction module to obtain the i+1-th first feature map, and when i+1=n, the i+1-th first feature map output by the i+1-th feature extraction module is used as the key point detection feature map of the image to be detected, i=1,2,3,...,n, where n is the total number of feature extraction modules; For the (i+1)th feature extraction module among the multiple feature extraction modules, wherein the (i+1)th feature extraction module includes a downsampling layer, a feature extraction layer, and an image stitching layer arranged in sequence according to an image processing direction; The downsampling layer in the (i+1)th feature extraction module is used to downsample the feature image output by the (i)th feature extraction module to obtain a first sampled image, and the size of the first sampled image output by the downsampling layer in the (i+1)th feature extraction module is half the size of the feature image output by the (i)th feature extraction module; The feature extraction layer in the (i+1)th feature extraction module is used to perform convolution processing and copy processing on the first sampled image to obtain a second feature map and a copy image; The image stitching layer in the (i+1)th feature extraction module is used to stitch the second feature map and the copied image to obtain an (i+1)th first feature map, wherein the (i+1)th first feature map has the same size as the second feature map and has a number of channels that is the sum of the number of channels of the second feature map and the number of channels of the copied image; The key point detection network is used to perform key point detection on the key point detection feature map to obtain the key point coordinates of the two wheels and the key point coordinates of the seat on the bicycle to be detected.

8. An electronic device, characterized in that: include: A memory, a processor, and a transceiver that are sequentially communicatively connected, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the shared bicycle rollover detection method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions are executed on a computer, the shared bicycle rollover detection method according to any one of claims 1 to 6 is executed.

Images