Medicine bottle label content identification method based on multiple cameras and YOLOv8

Abstract

The invention relates to the technical field of computer vision, image recognition and intelligent medicine management, in particular to a medicine bottle label content recognition method based on multiple cameras and YOLOv8. The method at least comprises the following steps: S1, deploying a medicine bottle label generation system, and generating a label; s2, multi-camera image acquisition and preprocessing; s3, carrying out chessboard calibration and space positioning; s4, medicine bottle label detection and label character recognition and structured analysis; and S5, system integration and application. According to the invention, through combination of multi-camera and multi-angle acquisition and checkerboard calibration positioning and combination with YOLOv8 label detection and OCR identification, high precision, high efficiency, end-to-end automation and system integration of medicine bottle label identification are realized, the defects of precision, efficiency, environmental adaptability and management integration in the prior art are overcome, and the system is suitable for popularization and application. The method has obvious technical advantages and practical value.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, image recognition, and intelligent drug management, specifically to a method for capturing, detecting, and recognizing drug bottle labels using a deep learning model and multiple cameras, and particularly to a method for recognizing drug bottle label content based on multiple cameras and YOLOv8. Background Technology

[0002] In modern pharmaceutical management and drug distribution, the identification of medicine bottle labels is a crucial component of drug management, warehousing and inventory control, and intelligent delivery. Medicine bottle labels typically contain key information such as the drug name, dosage, batch number, and expiration date, serving as an important basis for ensuring drug safety and compliance.

[0003] The existing medicine bottle label recognition process generally involves the following steps: First, images of the medicine bottle are acquired using a camera or scanning device. Then, image quality is improved through image preprocessing (such as grayscale conversion, denoising, binarization, and perspective correction). Next, traditional image processing algorithms (such as edge detection and template matching) or deep learning-based object detection models (such as Faster R-CNN and YOLOv5) are used to locate the label area. Subsequently, OCR technology is used to convert the text in the label area into editable text. Finally, the recognition results are post-processed, including denoising, format correction, and structured parsing, and the results are imported into a drug management system or used for warehouse inventory.

[0004] This traditional model has the following problems:

[0005] Label diversity makes identification difficult: medicine bottle sizes, label fonts, colors and layouts vary greatly, and traditional image processing algorithms are poor at adapting to this diversity, making it easy to miss or misdetect.

[0006] Sensitive to lighting and shooting angle: In real-world scenarios, medicine bottles may be tilted, rotated, reflective, or unevenly lit, leading to a decrease in OCR recognition accuracy.

[0007] Manual or semi-automatic operation is inefficient: In scenarios involving the processing of large quantities of medicines, the recognition process relying on manual review or traditional OCR is time-consuming and cannot meet real-time requirements.

[0008] High error rate affects management safety: Identification errors may lead to inaccurate drug information entry, affecting inventory management, drug dispensing and traceability, and posing certain safety risks.

[0009] System integration difficulties: The multi-step sequential identification process is difficult to automate to the end, resulting in high integration costs with warehouse management systems, logistics systems or intelligent inventory systems.

[0010] While some studies have attempted to improve recognition accuracy through deep learning models or improved OCR methods, issues such as unstable accuracy and poor real-time performance remain when dealing with multiple medicine bottles, various scenarios, and complex lighting conditions. Therefore, designing a high-precision, high-efficiency method for recognizing medicine bottle label content that can adapt to complex scenarios has become a pressing technical challenge for pharmaceutical management systems.

[0011] In view of this, the present invention aims to provide a method for recognizing the content of medicine bottle labels based on multiple cameras and YOLOv8, which has the characteristics of high content recognition accuracy and high efficiency, and can adapt to complex scenarios. Summary of the Invention

[0012] To address the shortcomings of existing technologies, a method for recognizing medicine bottle label content based on multiple cameras and YOLOv8 is provided. This method features high content recognition accuracy and efficiency, and can adapt to complex scenarios.

[0013] To solve the above problems, the technical solution of the present invention is as follows:

[0014] A method for recognizing the content of medicine bottle labels based on multiple cameras and YOLOv8 includes at least the following steps:

[0015] S1, Deployment of the medicine bottle label generation system and label generation: Deploy the medicine bottle label generation system locally. After accessing the system, a new label is generated for each medicine bottle by scanning the barcode. The label has a unique identifier, and each medicine bottle is bound to its unique label identifier. The system establishes a correspondence between the label and the barcode. The generated labels are mainly affixed to the medicine bottles.

[0016] S2, Multi-camera image acquisition and preprocessing: Deploy multiple cameras to acquire images of the medicine bottle from different angles, improve the recognizability of the label area through image preprocessing, expand the training samples through data augmentation, and improve the model's adaptability to different lighting, angles and materials;

[0017] S3, Chessboard calibration and spatial positioning: The chessboard calibration method is used to calibrate the camera, obtain intrinsic and extrinsic parameter information, realize the spatial coordinate alignment of multiple cameras, perform three-dimensional positioning of the detected label area, generate corresponding coordinates for each medicine bottle, and combine multiple camera angles to select the recognition result with the highest accuracy.

[0018] S4, Medicine Bottle Label Detection and Label Text Recognition and Structured Analysis: The YOLOv8 model is used to perform end-to-end detection on the images captured by each camera to locate the medicine bottle label area; OCR text recognition is performed on the detected label area to convert the text information in the image into editable text;

[0019] S5, System Integration and Application: Outputs the identification results of the tag content corresponding to the unique identifier to the drug management system or warehouse system to realize automated information updates.

[0020] As an improvement to the medicine bottle label content recognition method based on multiple cameras and YOLOv8 of the present invention, in S1, the barcode of each medicine bottle is scanned by a barcode scanner. The barcode is composed of a combination of English letters and numbers. The new label is affixed to the bottle cap for easy identification and management.

[0021] As an improvement to the medicine bottle label content recognition method based on multiple cameras and YOLOv8 of the present invention, in S2, the camera is located at the upper angle of the front, back, left and right sides inside the medicine cabinet.

[0022] As an improvement to the medicine bottle label content recognition method based on multiple cameras and YOLOv8 in this invention, in S2, image preprocessing includes denoising, perspective correction and contrast enhancement. Specifically, denoising involves using grayscale conversion, filtering denoising or morphological operations to dynamically adjust the size of each parameter.

[0023] Perspective correction specifically includes the following steps: (1) Detect the four corner points of the label; (2) Perform perspective transformation on the four points;

[0024] Contrast enhancement specifically involves using grayscale images, enhancing brightness / contrast, and adaptive histogram equalization to dynamically adjust various parameters.

[0025] As an improvement to the multi-camera and YOLOv8-based medicine bottle label content recognition method of this invention, in S2, the method of expanding the training samples using data augmentation to improve the model's adaptability to different lighting, angles, and materials is as follows:

[0026] (1) Illumination enhancement: Randomly adjust the brightness, contrast and saturation of the image to simulate different ambient light conditions, so that the model can still accurately recognize text under insufficient or excessive light conditions.

[0027] (2) Rotation and perspective transformation: Randomly rotate, affine or perspective transform the image to enhance the model’s adaptability to label tilt, shooting angle changes or shooting position deviations.

[0028] (3) Noise and blur processing: Add Gaussian noise, salt and pepper noise or light blur to the image to simulate noise interference or camera shake that may occur during shooting.

[0029] (4) Scaling, cropping and flipping: Randomly scale, crop and flip the label area horizontally / vertically so that the model can handle text of different sizes, partially occluded or reversed display.

[0030] (5) Material and color perturbation: Randomly adjust the text color, background texture or label material to improve the model's ability to recognize diverse printing methods or label materials.

[0031] As an improvement to the medicine bottle label content recognition method based on multiple cameras and YOLOv8 in this invention, in S3, the method of calibrating the cameras using a checkerboard calibration method to obtain intrinsic and extrinsic parameter information and achieve spatial coordinate alignment of multiple cameras is as follows:

[0032] (1) Prepare the calibration board: Use a checkerboard calibration board of known size and record the physical size of each square;

[0033] (2) Acquire calibration images: Take multiple chessboard images from different angles and distances for each camera to ensure that the chessboard covers different positions and directions in the images, and ensure sufficient calibration.

[0034] (3) Corner detection: Use OpenCV's findChessboardCorners to detect chessboard corners;

[0035] (4) Camera calibration: Use cv2.calibrateCamera to calculate the camera intrinsic parameter matrix K and distortion coefficients D to obtain the intrinsic parameters:

[0036] (5) Multi-camera extrinsic calibration: Multiple cameras are used to take pictures on the same chessboard, and the chessboard position in each camera coordinate system is obtained. The pose transformation matrix between cameras is solved by cv2.solvePnP or manually. The coordinate systems of each camera are aligned to a unified world coordinate system. The R and T of different cameras are uniformly transformed to achieve the spatial coordinate alignment of multiple cameras.

[0037] As an improvement to the multi-camera and YOLOv8-based medicine bottle label content recognition method of this invention, in S3, the detected label area is three-dimensionally located, corresponding coordinates are generated for each medicine bottle, and the method for selecting the recognition result with the highest accuracy by combining the angles of multiple cameras is as follows:

[0038] (1) Obtain the two-dimensional detection results of the labels: Use the YOLOv8 object detection model to obtain the label bounding boxes under the view of each camera and record each bounding box.

[0039] (2) Pixel coordinates to camera coordinates: Using the intrinsic parameter matrix K and distortion correction parameter D of each camera, the center point (u,v) of the bounding box is projected as the direction of light in the camera coordinate system;

[0040] (3) Combine extrinsic parameters for world coordinate mapping: use the camera extrinsic parameters (R,T) to map the light rays in the camera coordinate system to the world coordinate system, and then find the intersection of the light rays with the known height of the medicine bottle or the label plane to obtain the three-dimensional coordinates;

[0041] (4) Multi-camera fusion positioning: Fusion of 3D points from different cameras on the same medicine bottle: Collect the 3D coordinates and corresponding detection confidence scores calculated by each camera; Use weighted average or minimum reprojection error method to optimize the 3D coordinates to minimize the total error;

[0042] (5) Select the best recognition result: For each medicine bottle, the detection results of multiple cameras may differ. Based on the detection confidence, projection reprojection error and three-dimensional position deviation, the result with the highest accuracy is selected as the final label recognition result.

[0043] As an improvement to the medicine bottle label content recognition method based on multiple cameras and YOLOv8 of the present invention, the specific method of S4 is as follows:

[0044] (1) YOLO detection of label regions: YOLOv8 is used to detect the label regions in the images captured by each camera to obtain the bounding box of each label;

[0045] (2) OCR recognition: Perform OCR on each detection box.

[0046] (3) Collection of multi-angle detection results: For the same medicine bottle, multiple OCR results are obtained from different angles or different cameras;

[0047] (4) Selection of three-dimensional projection and minimum reprojection error: Map the center point of each detection box to three-dimensional space, perform preliminary three-dimensional point fusion, and calculate the weighted average or other preliminary fusion points for multiple detection boxes of the same medicine bottle;

[0048] (5) Minimum reprojection error selection: The fused 3D points are reprojected onto each camera image plane. The OCR text result with the smallest total reprojection error is selected as the final medicine bottle ID;

[0049] (6) Output the final result: Select the most reliable OCR text and obtain the three-dimensional coordinates.

[0050] As an improvement to the medicine bottle label content recognition method based on multiple cameras and YOLOv8 of the present invention, the specific method of S5 is as follows: the recognized medicine bottle number is transmitted to the system in real time through the interface, and the system outputs the corresponding medicine bottle information through the associated information.

[0051] In summary, this invention addresses the shortcomings of existing medicine bottle label recognition methods in terms of accuracy, efficiency, and environmental adaptability. It proposes a medicine bottle label content recognition method based on multiple cameras and YOLOv8, combined with chessboard calibration for spatial positioning, achieving end-to-end high-precision recognition. Specifically, this invention aims to address several technical deficiencies in existing medicine bottle label recognition methods, focusing on solving the following technical problems:

[0052] (1) Improve the accuracy and robustness of medicine bottle label detection.

[0053] Existing methods are prone to missing or misdetecting labels when faced with differences in bottle size, label font, color, layout, and lighting conditions.

[0054] How to design a method that utilizes multiple cameras and multi-angle data acquisition, combined with chessboard calibration for spatial positioning, to achieve high-precision and stable detection of medicine bottle labels in complex scenarios is a technical problem that urgently needs to be solved.

[0055] (2) Improve the accuracy and efficiency of label text recognition

[0056] Traditional OCR has limited effectiveness in recognizing label text that is tilted, reflective, has small font size, or is laid out in multiple lines or columns, and is prone to recognition errors.

[0057] The key technical problem that this invention needs to solve is how to perform high-precision OCR recognition on the detected multi-angle label regions and quickly convert the results into structured information.

[0058] (3) Achieve end-to-end automated processing and real-time identification

[0059] Existing processes typically rely on multiple steps in sequence (image acquisition → preprocessing → label detection → OCR → postprocessing), resulting in low processing efficiency and difficulty in meeting the needs of large-scale or real-time recognition.

[0060] The technical problem this invention aims to solve is how to build an end-to-end automated system that supports multi-camera image acquisition, label detection, text recognition, and structured output to achieve real-time medicine bottle label recognition.

[0061] (4) Reduce the system's dependence on shooting conditions and environment.

[0062] Traditional methods are highly sensitive to lighting, shooting angle, reflection, and label material, resulting in unstable recognition accuracy.

[0063] The key technical problem of this invention is how to design a method that can maintain a high accuracy rate for medicine bottle label recognition under different lighting, angle and material conditions, while combining multi-angle data fusion to improve overall robustness.

[0064] (5) Achieve structuring and system integration of recognition results

[0065] Existing methods cannot directly and seamlessly integrate the identification results with drug management systems, warehouse inventory systems, etc.

[0066] How to structure the results of multi-camera, multi-angle detection, and OCR recognition, and facilitate integration with enterprise management systems to automate drug management, is a key technical problem that this invention aims to solve. Compared with existing drug bottle label recognition technologies, the drug bottle label content recognition method based on multi-camera, YOLOv8, and checkerboard calibration proposed in this patent application has the following significant advantages:

[0067] First, improve detection and recognition accuracy: Utilize YOLOv8 for end-to-end label detection on multi-angle images, and combine it with deep learning OCR to achieve high-precision recognition of medicine bottle labels in complex scenarios. The combination of multi-camera acquisition and chessboard calibration spatial positioning effectively addresses the diversity of labels (font, color, layout) as well as changes in lighting, reflection, and shooting angle, improving the stability and reliability of detection and recognition.

[0068] Secondly, it improves processing efficiency and automation, achieving end-to-end automation from multi-angle image acquisition, label detection, OCR recognition to structured information output, eliminating the need for manual intervention and significantly reducing operation time. It supports batch and real-time recognition, meeting the needs of large-scale drug management and warehousing scenarios and improving overall work efficiency.

[0069] Third, the method enhances environmental adaptability, maintaining high recognition accuracy for medicine bottles of different materials, under different lighting conditions, and at different shooting angles, reducing the impact of environmental factors on recognition performance. Data augmentation, multi-camera information fusion, and model optimization strategies further improve the model's robustness in complex scenarios.

[0070] Fourth, it enables structured information output and system integration, directly converting identification results into structured data, facilitating integration with drug management systems, warehousing systems, or intelligent distribution systems to achieve automated information management. This improves the security and accuracy of drug management and reduces the risks of manual review and operational errors.

[0071] Fifth, it reduces enterprise management costs. Automated identification reduces reliance on manual labor, lowering labor costs and operational risks. It also improves the efficiency of drug distribution, inventory, and dispensing, optimizing enterprise operations and management.

[0072] In summary, this invention achieves high precision, high efficiency, end-to-end automation, and system integrability for medicine bottle label recognition by combining multi-camera, multi-angle acquisition with chessboard calibration and positioning, and integrating it with YOLOv8 label detection and OCR recognition. It overcomes the shortcomings of existing technologies in terms of accuracy, efficiency, environmental adaptability, and management integration, and has significant technical advantages and practical value. Attached Figure Description

[0073] Figure 1 This is a flowchart of the end-to-end process in this invention.

[0074] Figure 2 This is a flowchart of the image preprocessing sub-process in this invention.

[0075] Figure 3 This is a flowchart of the chessboard calibration and spatial positioning process in this invention.

[0076] Figure 4 This is a flowchart of the multi-camera OCR fusion and minimum error selection process in this invention.

[0077] Figure 5 This is a flowchart of system integration and structured output in this invention. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0079] like Figure 1-5 As shown, this invention addresses the shortcomings of existing medicine bottle label recognition methods in terms of accuracy, efficiency, and environmental adaptability. It proposes a medicine bottle label content recognition method based on multiple cameras and YOLOv8, combined with chessboard calibration for spatial positioning, achieving end-to-end high-precision recognition. Its core technical solution includes:

[0080] I. Deployment of the medicine bottle label generation system and label generation

[0081] A medicine bottle label generation system is deployed locally, along with a barcode scanner. After connecting to the system, a new, simplified label is generated for each medicine bottle by scanning its barcode. This label has a unique identifier, which can be a combination of letters and numbers. Each medicine bottle is uniquely linked to its label's unique identifier. The system establishes a mapping between labels and barcodes, while also pre-maintaining a mapping between barcodes and detailed medicine bottle information. The generated labels are primarily affixed to the bottle cap for easy identification and management.

[0082] II. Multi-camera image acquisition and preprocessing

[0083] 1. Deploy multiple cameras to capture images of the medicine bottle from different angles, covering the bottle cap position from multiple perspectives. Specifically, place cameras at the upper corners of the front, back, left, and right sides inside the medicine cabinet.

[0084] 2. Improve the recognizability of the label area through image preprocessing (denoising, perspective correction, contrast enhancement, etc.). Denoising specifically involves using grayscale conversion, filtering for noise reduction, morphological operations (removing small noise points), etc., and dynamically adjusting the values ​​of each parameter until a better effect is achieved. Perspective correction specifically involves (1) detecting the four corner points of the label (which can be done through edge detection + contour search); (2) performing perspective transformation on the four points. Contrast enhancement specifically involves using grayscale images, enhancing brightness / contrast, adaptive histogram equalization, and dynamically adjusting each parameter until a better effect is achieved.

[0085] 3. Expand the training samples using data augmentation to improve the model's adaptability to different lighting conditions, angles, and materials. Specifically:

[0086] (1) Illumination enhancement: Randomly adjust the brightness, contrast and saturation of the image to simulate different ambient light conditions, so that the model can still accurately recognize text under insufficient or excessive light conditions.

[0087] (2) Rotation and perspective transformation: Randomly rotate, affine or perspective transform the image to enhance the model’s adaptability to label tilt, shooting angle changes or shooting position deviations.

[0088] (3) Noise and blur processing: Gaussian noise, salt and pepper noise or mild blur are added to the image to simulate noise interference or camera shake that may occur during shooting.

[0089] (4) Scaling, cropping and flipping: Randomly scale, crop and flip the label area horizontally / vertically so that the model can handle text of different sizes, partially obscured or reversed.

[0090] (5) Material and color perturbation: Randomly adjust the text color, background texture or label material to improve the model's ability to recognize diverse printing methods or label materials.

[0091] III. Chessboard calibration and spatial positioning

[0092] 1. Use the chessboard calibration method to calibrate the cameras, obtain intrinsic and extrinsic parameter information, and achieve spatial coordinate alignment of multiple cameras. Specifically:

[0093] (1) Prepare the calibration board: Use a checkerboard calibration board of known size (such as 9×6 or 8×5 grids) and record the physical dimensions of each grid (unit: mm or cm). The larger the calibration board and the more grids, the higher the calibration accuracy.

[0094] (2) Acquire calibration images: Take multiple images of the chessboard from different angles and distances for each camera (generally more than 10-20 images). Ensure that the chessboard covers different positions and directions in the images to ensure sufficient calibration.

[0095] (3) Corner detection: Use OpenCV's findChessboardCorners to detect chessboard corners:

[0096] (4) Camera calibration (obtaining intrinsic parameters): Use cv2.calibrateCamera to calculate the camera intrinsic parameter matrix K and distortion coefficients D:

[0097] obj_points = [] # World coordinates img_points = [] # Image corner points

[0098] # Generate chessboard world coordinates objp=np.zeros((pattern_size[0]*pattern_size[1],3),np.float32)

[0099] objp[:,:2]=np.mgrid[0:pattern_size[0],0:pattern_size[1]].T.reshape(-1,2)*square_size_mm

[0100] # Fill obj_points and img_points for each image

[0101] obj_points.append(objp)

[0102] img_points.append(corners_subpix)

[0103] ret,K,D,rvecs,tvecs=cv2.calibrateCamera(obj_points,img_points,gray.shape[::-1],None,None)

[0104] K: Camera intrinsic parameter matrix, including focal lengths fx, fy and principal points cx, cy

[0105] D: Distortion coefficient (radial, tangential)

[0106] rvecs, tvecs: The rotation vector and displacement vector of each chessboard.

[0107] (5) Multi-camera extrinsic calibration (spatial coordinate alignment): Four cameras are used to capture images on the same chessboard, obtaining the chessboard position in each camera's coordinate system. The pose transformation matrix between cameras is solved using cv2.solvePnP or manually, aligning the coordinate systems of each camera to a unified world coordinate system.

[0108] ret,rvec,tvec=cv2.solvePnP(objp,corners_subpix,K,D)

[0109] R,_=cv2.Rodrigues(rvec) # Converts the rotation vector to a rotation matrix

[0110] T = tvec # Translation matrix

[0111] By uniformly transforming the R and T coordinates of different cameras, spatial coordinate alignment of multiple cameras can be achieved.

[0112] 2. Perform 3D localization on the detected label area to eliminate detection errors caused by angular deviations and improve multi-angle recognition accuracy. Generate corresponding coordinates for each medicine bottle and select the recognition result with the highest accuracy by combining multiple camera angles. Specifically:

[0113] (1) Obtaining 2D label detection results: The YOLOv8 object detection model is used to obtain the label bounding boxes from each camera's viewpoint. Record the bounding boxes for each bounding box:

[0114] Top-left corner coordinates (x_min, y_min)

[0115] The coordinates of the bottom right corner are (x_max, y_max).

[0116] Detect confidence score

[0117] (2) Pixel coordinates to camera coordinates: Using the intrinsic parameter matrix K and distortion correction parameter D of each camera, the center point (u,v) of the bounding box is projected as the direction of light in the camera coordinate system.

[0118] (3) Combine extrinsic parameters for world coordinate mapping: use the camera extrinsic parameters (R,T) to map the light rays in the camera coordinate system to the world coordinate system, and then find the intersection of the light rays with the known height of the medicine bottle or the label plane to obtain the three-dimensional coordinates.

[0119] (4) Multi-camera fusion localization: For the same medicine bottle, 3D points from different cameras are fused: the 3D coordinates calculated by each camera and the corresponding detection confidence are collected. The 3D coordinates are optimized using the weighted average or minimum reprojection error method to minimize the total error.

[0120] (5) Select the best recognition result: For each medicine bottle, the detection results from multiple cameras may differ. Based on the detection confidence, projection reprojection error, and 3D position deviation, select the result with the highest accuracy as the final label recognition result. A possible approach is to combine the following logic: If the confidence is low, try weighted fusion of multiple cameras. If the confidence is high and the projection error is small, directly use the result from that camera.

[0121] IV. Medicine Bottle Label Detection, Label Text Recognition and Structured Analysis

[0122] The YOLOv8 model is used to perform end-to-end detection on the images captured by each camera to locate the label area on the medicine bottle.

[0123] The detected label areas are subjected to OCR text recognition to convert the text information in the image into editable text.

[0124] Specifically:

[0125] (1) YOLO detection of label regions: YOLO is used to detect the label regions in the images captured by each camera, and the bounding box (x_min, y_min, x_max, y_max) of each label is obtained.

[0126] (2) OCR recognition: Perform OCR on each detection box.

[0127] (3) Collection of multi-angle detection results: For the same medicine bottle, multiple OCR results (ocr_results) are obtained from different angles or different cameras:

[0128]

[0129] (4) Selection of 3D projection and minimum reprojection error: Map the center point of each detection box to 3D space, perform preliminary 3D point fusion, and calculate the weighted average or other preliminary fusion points for multiple detection boxes of the same medicine bottle. (5) Selection of minimum reprojection error: Reproject the fused 3D points onto the image plane of each camera. Select the OCR text result with the smallest total reprojection error as the final medicine bottle ID.

[0130] (6) Output the final result: Select the most reliable OCR text and obtain the three-dimensional coordinates: final_bottle_id = best_result

[0131] final_3d_position=fused_point

[0132] V. System Integration and Application

[0133] The unique identifier is output to the drug management system or warehouse system to automate information updates. Specifically, the identified bottle number is transmitted to the system in real time via an interface, and the system outputs the usage status of the corresponding bottle based on associated information. Real-time and batch identification modes are supported, improving drug management efficiency and security.

[0134] In summary, this invention achieves high precision, high efficiency, end-to-end automation, and system integrability for medicine bottle label recognition by combining multi-camera, multi-angle acquisition with chessboard calibration and positioning, and integrating it with YOLOv8 label detection and OCR recognition. It overcomes the shortcomings of existing technologies in terms of accuracy, efficiency, environmental adaptability, and management integration, and has significant technical advantages and practical value.

[0135] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A method for recognizing the content of medicine bottle labels based on multiple cameras and YOLOv8, characterized in that, It should include at least the following steps: S1, Deployment of the medicine bottle label generation system and label generation: Deploy the medicine bottle label generation system locally. After accessing the system, a new label is generated for each medicine bottle by scanning the barcode. The label has a unique identifier, and each medicine bottle is bound to its unique label identifier. The system establishes a correspondence between the label and the barcode. The generated labels are mainly affixed to the medicine bottles. S2, Multi-camera image acquisition and preprocessing: Deploy multiple cameras to acquire images of the medicine bottle from different angles, improve the recognizability of the label area through image preprocessing, expand the training samples through data augmentation, and improve the model's adaptability to different lighting, angles and materials; S3, Chessboard calibration and spatial positioning: The chessboard calibration method is used to calibrate the camera, obtain intrinsic and extrinsic parameter information, realize the spatial coordinate alignment of multiple cameras, perform three-dimensional positioning of the detected label area, generate corresponding coordinates for each medicine bottle, and combine multiple camera angles to select the recognition result with the highest accuracy. S4, Medicine Bottle Label Detection and Label Text Recognition and Structured Analysis: The YOLOv8 model is used to perform end-to-end detection on the images captured by each camera to locate the medicine bottle label area; OCR text recognition is performed on the detected label area to convert the text information in the image into editable text; S5, System Integration and Application: Outputs the identification results of the tag content corresponding to the unique identifier to the drug management system or warehouse system to realize automated information updates.

2. The method for recognizing medicine bottle label content based on multiple cameras and YOLOv8 according to claim 1, characterized in that: In S1, the barcode of each medicine bottle is scanned by a barcode scanner. The barcode consists of a combination of English letters and numbers. The new label is affixed to the bottle cap for easy identification and management.

3. The method for recognizing medicine bottle label content based on multiple cameras and YOLOv8 according to claim 1, characterized in that: In S2, the camera is located at the upper corner of the front, back, left, and right sides inside the medicine cabinet.

4. The method for recognizing medicine bottle label content based on multiple cameras and YOLOv8 according to claim 1, characterized in that: In S2, image preprocessing includes denoising, perspective correction, and contrast enhancement. Specifically, denoising involves using grayscale conversion, filtering, or morphological operations to dynamically adjust the values ​​of various parameters. Perspective correction specifically includes the following steps: (1) Detect the four corner points of the label; (2) Perform perspective transformation on the four points; Contrast enhancement specifically involves using grayscale images, enhancing brightness / contrast, and adaptive histogram equalization to dynamically adjust various parameters.

5. The method for recognizing medicine bottle label content based on multiple cameras and YOLOv8 according to claim 1, characterized in that: In S2, the method of using data augmentation to expand the training samples and improve the model's adaptability to different lighting, angles, and materials is as follows: (1) Illumination enhancement: Randomly adjust the brightness, contrast and saturation of the image to simulate different ambient light conditions, so that the model can still accurately recognize text under insufficient or excessive light conditions. (2) Rotation and perspective transformation: Randomly rotate, affine or perspective transform the image to enhance the model’s adaptability to label tilt, shooting angle changes or shooting position deviations. (3) Noise and blur processing: Add Gaussian noise, salt and pepper noise or light blur to the image to simulate noise interference or camera shake that may occur during shooting. (4) Scaling, cropping and flipping: Randomly scale, crop and flip the label area horizontally / vertically so that the model can handle text of different sizes, partially occluded or reversed display. (5) Material and color perturbation: Randomly adjust the text color, background texture or label material to improve the model's ability to recognize diverse printing methods or label materials.

6. The method for recognizing medicine bottle label content based on multiple cameras and YOLOv8 according to claim 1, characterized in that: In S3, the checkerboard calibration method is used to calibrate the cameras, obtain intrinsic and extrinsic parameter information, and achieve spatial coordinate alignment of multiple cameras. The specific method is as follows: (1) Prepare the calibration board: Use a checkerboard calibration board of known size and record the physical size of each square; (2) Acquire calibration images: Take multiple chessboard images from different angles and distances for each camera to ensure that the chessboard covers different positions and directions in the images, and ensure sufficient calibration. (3) Corner detection: Use OpenCV's findChessboardCorners to detect chessboard corners; (4) Camera calibration: Use cv2.calibrateCamera to calculate the camera intrinsic parameter matrix K and distortion coefficients D to obtain the intrinsic parameters: (5) Multi-camera extrinsic calibration: Multiple cameras are used to take pictures on the same chessboard, and the chessboard position in each camera coordinate system is obtained. The pose transformation matrix between cameras is solved by cv2.solvePnP or manually. The coordinate systems of each camera are aligned to a unified world coordinate system. The R and T of different cameras are uniformly transformed to achieve the spatial coordinate alignment of multiple cameras.

7. The method for recognizing medicine bottle label content based on multiple cameras and YOLOv8 according to claim 1, characterized in that: In S3, the detected label area is 3D localized, generating corresponding coordinates for each medicine bottle. The method for selecting the most accurate recognition result by combining multiple camera angles is as follows: (1) Obtain the two-dimensional detection results of the labels: Use the YOLOv8 object detection model to obtain the label bounding boxes under the view of each camera and record each bounding box. (2) Pixel coordinates to camera coordinates: Using the intrinsic parameter matrix K and distortion correction parameter D of each camera, the center point (u,v) of the bounding box is projected as the direction of light in the camera coordinate system; (3) Combine extrinsic parameters for world coordinate mapping: use the camera extrinsic parameters (R,T) to map the light rays in the camera coordinate system to the world coordinate system, and then find the intersection of the light rays with the known height of the medicine bottle or the label plane to obtain the three-dimensional coordinates; (4) Multi-camera fusion positioning: Fusion of 3D points from different cameras on the same medicine bottle: Collect the 3D coordinates and corresponding detection confidence scores calculated by each camera; Use weighted average or minimum reprojection error method to optimize the 3D coordinates to minimize the total error; (5) Select the best recognition result: For each medicine bottle, the detection results of multiple cameras may differ. Based on the detection confidence, projection reprojection error and three-dimensional position deviation, the result with the highest accuracy is selected as the final label recognition result.

8. The method for recognizing medicine bottle label content based on multiple cameras and YOLOv8 according to claim 1, characterized in that, The specific method for S4 is as follows: (1) YOLO detection of label regions: YOLOv8 is used to detect the label regions in the images captured by each camera to obtain the bounding box of each label; (2) OCR recognition: Perform OCR on each detection box. (3) Collection of multi-angle detection results: For the same medicine bottle, multiple OCR results are obtained from different angles or different cameras; (4) Selection of three-dimensional projection and minimum reprojection error: Map the center point of each detection box to three-dimensional space, perform preliminary three-dimensional point fusion, and calculate the weighted average or other preliminary fusion points for multiple detection boxes of the same medicine bottle; (5) Minimum reprojection error selection: The fused 3D points are reprojected onto each camera image plane. The OCR text result with the smallest total reprojection error is selected as the final medicine bottle ID; (6) Output the final result: Select the most reliable OCR text and obtain the three-dimensional coordinates.

9. The method for recognizing medicine bottle label content based on multiple cameras and YOLOv8 according to claim 1, characterized in that, The specific method of S5 is as follows: the identified medicine bottle number is transmitted to the system in real time through the interface, and the system outputs the corresponding medicine bottle information through the associated information.

Images