Method for automatically identifying target and scoring

Abstract

The invention relates to the field of image recognition, in particular to a method for automatically recognizing an arrow target and scoring. The invention provides a method for automatically identifying an arrow target and scoring, which comprises the following steps of: acquiring a picture set, processing the picture set as a training set and a verification set, obtaining a pre-trained Caffe model, preferably constructing the spatial positions of the arrow target and an arrow ring with a calibration period, performing image identification by combining an OpenCV DNN module with the Caffe model, restoring a three-dimensional coordinate system of an arrow by adopting a spatial conversion technology, and automatically identifying and scoring the arrow target and the arrow ring. The method for automatically identifying the arrow target and scoring is provided, and the effects of identifying the arrow target and the arrow drop point in real time and calculating the score can be achieved.

Description

Technical Field

[0001] The invention relates to the field of image recognition, and in particular to a method for automatically recognizing and scoring an arrow target. Background Art

[0002] With the development of information technology and the increasing computing power of computers, image processing and efficient image recognition on mobile devices have become research hotspots. AI is used to replace manual work to complete highly repetitive tasks with low business difficulty. Especially in computer vision, as image recognition technology becomes more and more popular, it can replace manual work to complete functions such as target classification. However, there is a problem that image recognition requires high computing power for the device.

[0003] The Caffe model is a deep learning framework used in the field of artificial intelligence. It is well-known for its efficiency, flexibility, and modular design. Due to its characteristics, the Caffe model is suitable for research and application in fields such as computer vision, natural language processing, and language recognition.

[0004] OpenCV is an open source computer vision and learning software library. It is cross-platform and can run on multiple operating systems, including Linux, Windows, Android, MacOS and other operating systems. It supports multiple languages, including interfaces of languages ​​such as C++, Python, Java and MATLAB. It has rich algorithms and models, ranging from basic image processing operations to complex computer vision algorithms such as feature detection, image segmentation, motion tracking, etc. It has designed efficient algorithms to achieve real-time image and video processing. It supports the use of GPU accelerated computing, can achieve hardware acceleration, and has scalability.

[0005] In the OpenCV DNN module, it has an important position in the field of image recognition. The DNN module supports multiple deep learning frameworks, such as Caffe, TensorFlow, Torch, etc., allowing users to load pre-trained models and apply them to various image recognition tasks, including but not limited to image classification, object detection, semantic segmentation, etc. Image classification is a typical application of the DNN module. By using pre-trained deep neural network models, such as the GoogleNet model or the ResNet model, the DNN module can recognize and classify objects in images. For example, using the GoogleNet model, the DNN module can load the Caffe model data and realize the recognition of common image classifications. In addition, the DNN module also supports object detection in real-time video, and can process video streams and identify multiple objects in the video. Another important application of the DNN module is target detection, which can use models such as SSD (Single Shot Detector) to detect multiple objects in the image and mark them with boxes. The DNN module also supports inference on the GPU, which greatly improves the processing speed of image recognition tasks, making it suitable for application scenarios that require fast response. The ease of use and efficiency of the OpenCVDNN module make it a very popular tool in the field of image recognition. It not only supports a variety of deep learning models and frameworks, but also provides compatibility with different devices and operating systems, including support for mobile platforms. Through the DNN module, developers can easily integrate deep learning technology into their image processing applications to achieve advanced image recognition functions.

[0006] One of the best things about the OpenCV DNN module is that it is highly optimized for CPU, giving it the ability to get fast inference even without a powerful GPU system, but it still puts a heavy burden on the device when deploying larger models on devices with limited computing power.

[0007] At present, the judgment of the score of arrows on the target in archery competitions mainly relies on manual scoring, which is based on the number of rings the arrow hits on the target. The closer the arrow is to the center of the bull's eye, the higher the number of rings obtained and the higher the score obtained, which involves issues such as labor costs, efficiency and speed. Summary of the invention

[0008] The purpose of the present invention is to reduce the equipment operation burden caused by converting spatial three-dimensional coordinates while running a Caffe model, to automatically identify targets and score archery, and to provide a method for automatically identifying targets and scoring.

[0009] The present invention provides a method for automatically identifying a target and scoring, comprising the following steps: Get the image set, including the training image set and the validation image set, which correspond to the training set and validation set of the Caffe model respectively; Performing sample training on the Caffe model using the picture set to obtain a pre-trained Caffe model; During the calibration period, the camera is used to collect images, and the pre-trained Caffe model is used to identify the position of the target plane, so as to construct the spatial position of the target and the arrow ring; Use OpenCV DNN module to load the pre-trained Caffe model; use the camera to collect real-time images; Using the pre-trained Caffe model to recognize the real-time image, obtain the positions of the target and the arrow, and during recognition, for the image with changes, perform spatial transformation on the newly recognized arrow, and obtain the spatial three-dimensional coordinates of the arrow through spatial transformation; The score is calculated by combining the spatial three-dimensional coordinates of the arrow and the position of the target plane.

[0010] A picture set is obtained as a sample to train a Caffe model, and the recognition ability of the Caffe model for the target is improved in a targeted manner, so that the recognition and scoring are more accurate; the Caffe model is sampled and trained using the picture set to obtain a pre-trained Caffe model as a basic model for automatically recognizing the target and scoring according to the present invention; the spatial position of the target and the arrow ring is preferentially constructed, and the target at the spatial position is no longer repeatedly recognized in subsequent recognition, thereby reducing the amount of calculation and allocating limited resources reasonably; the spatial three-dimensional coordinates of the target and the arrow are obtained as the basis for subsequent scoring; when recognizing the real-time image, a spatial transformation is performed on the newly recognized arrow for the picture with changes in recognition, so as to save computing resources; the score is calculated by combining the spatial three-dimensional coordinates of the arrow and the position of the target plane, and the score of the archery result can be obtained.

[0011] Preferably, the acquired picture set has an object of a target, which requires bright lighting, no obstructions, and covers scenes under different viewing angles and different lighting conditions.

[0012] The different viewing angles are used to take photos from different directions around the target, up, down, left, and right; The different lightings simulate sunlight, cloudy days, incandescent lamps, and tungsten filament lamps.

[0013] The image set used for training needs to take into account environmental factors, such as lighting, occlusion, different viewing angles, etc. The images can be processed into LMDB format data through tools as input for the Caffe model.

[0014] Preferably, the use of the picture set to perform sample training on the Caffe model refers to converting the picture set into data input into the Caffe model, establishing a training set and a validation set for the recognition of targets and arrows, and using the training set and the validation set to train the Caffe model to obtain a pre-trained Caffe model.

[0015] The pre-trained Caffe model usually recognizes a variety of objects, most of which are not required by the present invention and will affect the recognition performance and efficiency of arrows and targets. The LMDB format data converted from the picture set collected as samples is used as a training sample for the pre-trained Caffe model, and the recognition capabilities of cylinders and circles are trained specifically. A training set and a verification set are established for target recognition to obtain the pre-trained Caffe model.

[0016] Preferably, during the calibration period, the camera needs to be calibrated before the program is started so that the position of the target is aligned with the camera to obtain a captured image, and the pre-trained Caffe model is used to recognize the captured image to obtain the position of the target plane and construct the spatial position of the target and the arrow ring.

[0017] Preferably, the image is collected by a camera, and the camera simulates human eye stereoscopic vision to construct a binocular imaging model.

[0018] For the pre-trained Caffe model, its image recognition can only return the coordinate positions of the target and the arrow, and cannot determine the arrow landing point based on the returned coordinates. The camera needs to include two cameras with equal focal lengths. The binocular distance of a person is known. The camera simulates the stereoscopic vision of the human eye to obtain the binocular imaging model, which can further restore the three-dimensional relationship between the target and the arrow, thereby determining the arrow landing point.

[0019] In actual shooting, the two cameras shoot from different angles. They are on the same z-axis and y-axis, but different x-axis. The difference between the two points can be calculated. Therefore, the position of the arrow and the target in the three-dimensional world can be obtained based on the acquired binocular imaging model.

[0020] Preferably, the recognition of the changed images refers to using the SSIM algorithm to calculate the image similarity between the current key frame and the previous key frame, and when the similarity is lower than a threshold, the current key frame is input into the pre-trained Caffe model to recognize the target and the arrow, which can save computing resources. Generally speaking, the threshold can be 0.9.

[0021] Preferably, the spatial transformation for the newly identified arrow means that not all arrows are identified, and during the identification process, all arrow positions returned by the pre-trained Caffe model are saved, and the arrows that are different from the saved positions of all arrows are newly identified, and spatial transformation is performed on the newly identified arrows.

[0022] In actual archery, arrows do not fall on the target continuously, and every frame is not recognized. Spatial conversion is performed on the newly recognized arrows, saving computing resources.

[0023] Preferably, the spatial three-dimensional coordinates of the arrow obtained by spatial transformation refer to setting the key frame where the newly identified arrow is located as a valid key frame, and using the pre-trained Caffe model to spatially transform the arrow in the valid key frame to obtain the spatial three-dimensional coordinates of the arrow, wherein the spatial three-dimensional coordinates of the arrow include the arrow starting point and the arrow ending point.

[0024] The three-dimensional coordinates of the target and arrow are obtained as the basis for subsequent scoring.

[0025] The method for obtaining the three-dimensional coordinates in space is: using the binocular distance, the focal length of the camera, and the principle of similar triangles, respectively detecting the arrows in the two images in the binocular imaging model, obtaining the pixel position of the arrow, and performing bit depth calculation to obtain the coordinate point of the arrow in three-dimensional space. Assuming that the known binocular distance is b, the focal length of the camera is f, and the parallax is d, the three-dimensional coordinates in space are z = f*b / d, x=b + xr*z / f, y=yr*z / f; wherein xr is the horizontal coordinate of the arrow located at the imaging of the right camera, and yr is the vertical coordinate of the right camera. The bit depth calculation is performed to calculate the distance of the object from the camera, and a coordinate system is established accordingly to obtain the three-dimensional coordinates in space.

[0026] Preferably, calculating the score by combining the spatial three-dimensional coordinates of the arrow and the position of the target plane means taking the distance from the starting point of the arrow to the ending point of the arrow as a vector, calculating the intersection of the vector and the target plane, and obtaining the arrow score.

[0027] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention provides a method for automatically identifying and scoring archery targets. The method obtains a picture set, performs sample training on a Caffe model, obtains a pre-trained Caffe model, uses the pre-trained Caffe model for image recognition, and has a calibration period, which can help improve the accuracy of archery image recognition.

[0028] 2. The present invention provides a method for automatically identifying and scoring an archery target, and the score is calculated based on the spatial three-dimensional coordinates of the arrow obtained from the recognition image and the position of the target plane, which can achieve the effect of obtaining the archery result score based on real-time image recognition.

[0029] 3. The present invention provides a method for automatically identifying and scoring targets. During identification, a spatial transformation is performed on a changed image and a newly identified arrow, thereby saving computing resources and reducing the burden on equipment operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 An example diagram of the structure for scoring recognition on Android.

[0031] Figure 2 Schematic diagram of binocular imaging for the human eye.

[0032] Figure 3 Schematic diagram of bit depth calculation. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with test examples and specific implementation methods. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0034] Example 1 A method for automatically identifying a target and scoring an arrow, this embodiment is run on an Android platform, such as Figure 1 As shown, the Android platform loads the OpenCV module and the archery result scoring module. The OpenCV module is equipped with the OpenCV DNN module and the Caffe model. The OpenCV DNN module loads the Caffe model. The Caffe model inputs the spatial three-dimensional coordinates of the arrow and the position of the target plane output by the recognition into the archery result scoring module.

[0035] Get the image set, including the training image set and the validation image set, which correspond to the training set and validation set of the Caffe model respectively; The picture set includes scenes of different viewing angles around the target in different directions, such as up, down, left, and right, and scenes of different lighting, including lighting simulating sunlight, cloudy days, incandescent lamps, and tungsten filament lamps.

[0036] In this embodiment, the picture set is processed to obtain LMDB format data, that is, Lightning Memory-Mapped Database format data, which is input into the Caffe model for sample training, and a training set and a validation set are established for target recognition to obtain a pre-trained Caffe model.

[0037] In this embodiment, a binocular camera is used. During the calibration period, the binocular camera needs to be calibrated before the program is started so that the position of the target is aligned with the binocular camera. The binocular camera collects real-time images to obtain a binocular imaging model. The spatial three-dimensional coordinates of the target are obtained through spatial transformation. The pre-trained Caffe model recognizes the binocular imaging model, obtains the target plane, and constructs the spatial position of the target and the arrow ring.

[0038] Use OpenCV DNN module to load the pre-trained Caffe model; use the camera to collect real-time images; To load the pre-trained Caffe model using OpenCV DNN, two files need to be loaded, one is the model, which contains the pre-trained weight file with the extension .caffemodel, and the other is the model architecture file with the extension .prototxt. The Caffe model will return the identified object id and its position coordinates together, and load the list of object names that the Caffe model can recognize into the memory so as to know what kind of object the Caffe model recognizes. Loading the model requires the pre-trained weight file (.caffemodel) and the model configuration file (.prototxt). Use the readNetf(config, model,framework) function to load the model, where config is the model configuration file, i.e., the .prototxt file; model is the pre-trained weight file; framework loads the framework name of the model framework. Use blobFromImage(image,scalefactor, size, mean, swapRB, crop, ddepth) to convert the image into LMDB format and input it into the Caffe model, where image is the input image, scalefactor represents the scaling factor (the default value is 1, indicating no scaling), size represents the size to which the image is adjusted, and mean is the normalized input, which is actually the average value of the RGB color channels of the image minus the average value.

[0039] The pre-trained Caffe model is used to identify the real-time image, and the positions of the target and the arrow are obtained. To identify the changed picture in the real-time image captured by the binocular camera, in this embodiment, the SSIM algorithm is used to calculate the picture similarity between the current key frame and the previous key frame. When the similarity is lower than the threshold value of 0.9, the current key frame is input into the pre-trained Caffe model to identify the target and the arrow. The newly identified arrow is spatially transformed, and the positions of all arrows returned by the pre-trained Caffe model are saved during the recognition process. The arrow that is different from the saved positions of all arrows is the newly identified arrow. The newly identified arrow is spatially transformed, and the key frame where the newly identified arrow is located is the valid key frame.

[0040] In this embodiment, a binocular camera with known equal internal focal length is used, that is, the binocular distance is known, and the three-dimensional coordinates of the object can be reconstructed by simulating the stereoscopic vision of the human eye. Figure 2 As shown in the figure, P is the image point, that is, the real position of the object in the three-dimensional space, Pl and Pr are the image points of the left and right cameras, Ol and Or are the positions of the left and right cameras, forming a binocular imaging model. The position of the physical point in the three-dimensional space is obtained through the position and posture of the two cameras and the projection points of the same physical point on the two images.

[0041] The arrow in the valid key frame is spatially transformed using the pre-trained Caffe model to obtain the three-dimensional spatial coordinates of the arrow, wherein the three-dimensional spatial coordinates of the arrow include an arrow starting point and an arrow ending point.

[0042] Using binocular distance, focal length and other parameters, combined with the principle of similar triangles, the arrows in the two images are detected to obtain the pixel position of the arrows. Substituting them into the bit depth calculation formula, the coordinates of the arrows in three-dimensional space can be obtained. Assuming that the binocular distance is b, the focal length of the camera is f, and the parallax is d, the three-dimensional spatial coordinates z = f*b / d, x=b + xr*z / f, y=yr*z / f; where xr is the horizontal coordinate of the arrow located at the right camera imaging, and yr is the vertical coordinate of the right camera. The bit depth calculation is as follows: Figure 3 As shown, the X-axis and the Z-axis are perpendicular to each other, the left camera and the right camera of the binocular camera are on both sides of the baseline, the image point is P = (x, z), b is the distance between the left camera and the right camera, f is the focal length of the camera, x is the vertical distance from the image point to the left camera, z is the horizontal distance between the image point and the camera, xl is the deviation of the image point in the left camera, and xr is the deviation of the image point in the right camera. The world coordinate system is built accordingly.

[0043] The arrow starting point to the arrow ending point is taken as a vector, and the intersection of the vector and the target plane is calculated to obtain the arrow score.

[0044] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatically identifying and scoring an archery target, characterized in that: The following steps are involved: Get the image set, including the training image set and the validation image set, which correspond to the training set and validation set of the Caffe model respectively; Performing sample training on the Caffe model using the picture set to obtain a pre-trained Caffe model; During the calibration period, the camera is used to collect images, and the pre-trained Caffe model is used to identify the position of the target plane, so as to construct the spatial position of the target and the arrow ring; Use OpenCV DNN module to load the pre-trained Caffe model; use the camera to collect real-time images; Using the pre-trained Caffe model to recognize the real-time image, obtain the positions of the target and the arrow, and during recognition, for the image with changes, perform spatial transformation on the newly recognized arrow, and obtain the spatial three-dimensional coordinates of the arrow through spatial transformation; The score is calculated by combining the spatial three-dimensional coordinates of the arrow and the position of the target plane.

2. The method for automatically identifying and scoring an archery target according to claim 1, characterized in that: The acquired picture set is an archery target, and requires bright lighting, no obstructions, and covers scenes under different viewing angles and different lighting conditions.

3. The method for automatically identifying and scoring an archery target according to claim 1, characterized in that: The sample training of the Caffe model using the picture set refers to converting the picture set into data input into the Caffe model, establishing a training set and a verification set for the recognition of targets and arrows, and using the training set and the verification set to train the Caffe model to obtain a pre-trained Caffe model.

4. The method for automatically identifying and scoring an archery target according to claim 1, characterized in that: During the calibration period, the camera needs to be calibrated before the program is started so that the position of the target is aligned with the camera to obtain a captured image. The captured image is recognized using the pre-trained Caffe model to obtain the position of the target plane and construct the spatial position of the target and the arrow ring.

5. The method for automatically identifying and scoring an archery target according to claim 1, characterized in that: The camera is used to collect images, and the camera simulates the stereoscopic vision of human eyes to build a binocular imaging model.

6. The method for automatically identifying and scoring an archery target according to claim 1, characterized in that: The method for identifying a changed picture refers to using the SSIM algorithm to calculate the picture similarity between the current key frame and the previous key frame. When the similarity is lower than a set threshold, the current key frame is input into the pre-trained Caffe model to identify the target and the arrow.

7. The method for automatically identifying and scoring an archery target according to claim 1, characterized in that: The spatial transformation for the newly identified arrows means that not all arrows are identified, and during the identification process, all arrow positions returned by the pre-trained Caffe model are saved. Arrows that are different from the saved positions of all arrows are newly identified, and spatial transformation is performed on the newly identified arrows.

8. The method for automatically identifying and scoring an archery target according to claim 1, characterized in that: The spatial three-dimensional coordinates of the arrow obtained by spatial transformation refer to setting the key frame where the newly identified arrow is located as a valid key frame, and using the pre-trained Caffe model to spatially transform the arrow in the valid key frame to obtain the spatial three-dimensional coordinates of the arrow, wherein the spatial three-dimensional coordinates of the arrow include the starting point of the arrow and the ending point of the arrow.

9. The method for automatically identifying and scoring an archery target according to claim 1, characterized in that: Calculating the score by combining the spatial three-dimensional coordinates of the arrow and the position of the target plane means taking the area from the arrow starting point to the arrow ending point as a vector, calculating the intersection of the vector and the target plane, and obtaining the arrow score.

Images