AI scene recognition photo generation method based on intelligent scene viewing equipment

Abstract

The invention discloses an AI scene recognition photo generation method based on intelligent scene viewing equipment, and the method comprises the steps: employing a camera as a picture collection end, employing a display screen as a picture output end, training an AI recognition model obtained based on a deep learning algorithm, and completing the automatic recognition of sceneries for the viewable sceneries of each scenic region; meanwhile, through a scenery selection algorithm, the scenery which most meets the expectation of the tourist is selected from a plurality of recognition results; then, through a chartlet position algorithm of the scenery description image-text, the description is attractive and easy to see, meanwhile, the scenery is not shielded, and finally multiple layers of elements are combined into a complete image. No extra artificial participation is needed in the whole process, the AI automatically generates the electronic postcard for the tourist, the background picture of the postcard is shot by the tourist, each tourist has different personalized contents, the electronic postcard can be downloaded to a mobile phone, the tourist can share the electronic postcard to the friend circle of the tourist, and the experience of the tourist is improved. And the personalized pursuit and sharing requirements of tourists are met.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for generating AI scene recognition photos based on intelligent viewing equipment. Background Art

[0002] Taking beautiful photos is a basic need for tourists when visiting scenic spots. In the past, the traditional way was that many scenic spots would select some beautiful photos of scenic spots, and add illustrated explanations to make postcards, which were sold to tourists as souvenirs.

[0003] However, the traditional postcard production method requires a lot of manpower, and the content obtained by each tourist is exactly the same, which can no longer meet the tourists' pursuit of personalization. Summary of the invention

[0004] The purpose of the present invention is to propose an AI scene recognition photo generation method based on an intelligent viewing device. Without any additional human participation, AI automatically generates electronic postcards for tourists, and the background picture of the postcard is taken by the tourists themselves. Each tourist has different personalized content. The electronic postcard can be downloaded to a mobile phone, and tourists can share it to their own circle of friends, satisfying the tourists' personalized pursuit and sharing needs.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] The present invention proposes an AI scene recognition photo generation method based on an intelligent scene viewing device, comprising the following steps:

[0007] S1, tourists use smart viewing devices to obtain photos;

[0008] S2. Use the AI ​​scene recognition model to identify scenic spots in the photos taken by tourists, including the tag name, confidence level and range coordinates of the scene in the picture; when the AI ​​scene recognition model recognizes multiple scenes, use the C-position analysis algorithm to locate a scene in the most central position as the final result of this recognition;

[0009] S3. Based on the final recognition result, AI recognition is used to automatically configure the photo with illustrated explanation content, LOGO and digital photo frame to generate an electronic postcard;

[0010] The intelligent viewing device at least includes a camera for image acquisition and a display screen capable of displaying the acquired images for tourists to browse. The device also has a core circuit board composed of a CPU, GPU or NPU processing chip to perform camera data processing.

[0011] Furthermore, the specific process of step S1 is:

[0012] The smart viewing device uses a camera as the image acquisition end and a display screen as the image output end. The human-computer interaction includes four buttons: zoom in, zoom out, take a photo, and AI function on / off. Tourists use the zoom in or zoom out button to adjust the image to a satisfactory optimal image position, and then click the photo button. At this time, the photo action triggers an AI recognition task, and at the same time, a background photo is obtained from the camera's video stream through a screenshot method, completing the tourist photo process.

[0013] Furthermore, the AI ​​scene recognition model in step S2 is a scene recognition model obtained by training at least 100 photos of each scene based on a deep learning algorithm, and the deep learning algorithm adopts the YOLOv8 algorithm.

[0014] Furthermore, the C-position analysis algorithm in step S2 is: compare the center point Xn coordinates of all recognized objects with the center point Xo coordinates of the screen resolution, and select the object with the smallest distance to the center point Xo coordinates of the screen resolution as the recognized scene.

[0015] Furthermore, the specific process of step S3 is: by selecting the tag name of the object, the description graphic information of the scenery corresponding to the tag name is filtered out from the device storage, and a map position selection algorithm is used to select a reasonable position from the left and right sides to place the scenery description graphic information according to the location of the scenery; at the same time, the scenic spot LOGO is pasted on the lower left corner of the entire picture, the current date is pasted on the lower right corner, a digital photo frame is added on the outermost side, and the image synthesis function in the ffmpeg image processing algorithm library is called to merge all elements hierarchically to finally generate an electronic postcard.

[0016] Furthermore, the image posting position selection algorithm is as follows: by default, the scene description graphic information is placed on the left side of the image; if the calculated scene range coordinates are biased to the left side of the image, the scene description graphic information is posted on the right side of the image.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] The AI ​​scene recognition photo generation method based on the intelligent scene viewing device provided by the present invention uses a camera as the picture acquisition end and a display screen as the picture output end. Based on the AI ​​recognition model for the viewable scenery of each scenic spot obtained by deep learning algorithm training, the scene is automatically recognized; at the same time, the scene selection algorithm is used to select the scene that best meets the expectations of tourists from multiple recognition results; then, the description is made beautiful, easy to read, and does not block the scenery through the mapping position algorithm of the scene description image and text; finally, the multi-layer elements are merged into a complete picture, and saved to the device end and the background cloud platform end for tourists to download and obtain. No additional manual participation is required throughout the process. The AI ​​automatically generates electronic postcards for tourists, and the background pictures of the postcards are taken by the tourists themselves. Each tourist has different personalized content. The electronic postcards can be downloaded to mobile phones, and tourists can share them to their own circle of friends, which meets the personalized pursuit and sharing needs of tourists. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0020] Figure 1 A flowchart of the steps of the AI ​​scene recognition photo generation method based on the intelligent viewing device provided by the present invention.

[0021] Figure 2 This is an example diagram of a tourist taking a photo provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the mapping position selection algorithm provided by the present invention.

[0023] Figure 4 The AI ​​scene recognition photo generation method based on the intelligent scene viewing device provided by the present invention obtains the effect picture of the electronic postcard. DETAILED DESCRIPTION

[0024] In order to better understand the technical solution, the method of the present invention is described in detail below with reference to the accompanying drawings.

[0025] The present invention proposes an AI scene recognition photo generation method based on intelligent viewing equipment, which is the best technical route optimized after a large number of tests and long-term research combined with tourists' usage habits. Figure 1 As shown, it mainly includes tourists taking photos, AI scene recognition, and AI Figure 3 The steps are as follows.

[0026] S1. Tourists take photos: Tourists use smart viewing devices to take photos.

[0027] The intelligent viewing device at least includes a camera for image acquisition and a display screen capable of displaying the acquired images for tourists to browse. The device also has a core circuit board composed of a CPU, GPU or NPU processing chip to perform camera data processing.

[0028] The smart viewing device uses a camera as the image acquisition end and a display screen as the image output end. The human-computer interaction includes four buttons: zoom in, zoom out, take a photo, and turn the AI ​​function on / off.

[0029] The specific process of tourists taking photos is as follows: tourists adjust the picture to a satisfactory optimal picture position through the zoom in or zoom out button, and then click the photo button. At this time, the photo action triggers an AI recognition task, and at the same time, a background photo is obtained from the video stream of the camera through the screenshot method, completing the tourist photo process.

[0030] S2. AI scene recognition: Use the AI ​​scene recognition model to identify scenic spots and scenery in photos taken by tourists.

[0031] Call the AI ​​scene recognition model, which is also trained based on the deep learning algorithm. It is necessary to train at least 100 photos for each scene to obtain the scene recognition model. The AI ​​scene recognition model should complete the recognition of the scene in the photos taken by tourists within 0.1 seconds, including the label name, confidence and range coordinates of the scene. When the AI ​​scene recognition model recognizes multiple scenes, the system locates a scene in the most central position through the C-position analysis algorithm as the final result of this recognition. The deep learning algorithm adopts the YOLOv8 algorithm. The C-position analysis algorithm is: compare the center point Xn coordinates of all recognized objects with the center point Xo coordinates of the screen resolution, and select the object with the smallest distance from the center point Xo coordinates of the screen resolution as the most recognized scene.

[0032] Take the smart viewing equipment deployed in Beigaofeng Scenic Area as an example:

[0033] 1) First, tourists take photos, which will accumulate a large number of pictures, which will be uploaded to the cloud server. Figure 2 shown.

[0034] 2) A batch data download function has been developed. Through this interface, you can fill in the device number and date of the data to be downloaded, and download the latest pictures from the cloud album to the local computer.

[0035] https: / / zhgj.7etec.top / server / jtzhgj / mine / downloadCloudPhoto? deviceId = device unique number & dateTime = download start date. Figure 2 shown.

[0036] 3) Filter the images in the cloud album for training materials and then compress them.

[0037] 4) Upload the new image to the annotation system via compressed file. The annotation system is developed based on WEB technology to facilitate the work of the annotator.

[0038] 5) Filter out the unlabeled images and label them.

[0039] 6) On the server side, use the YOLO algorithm command to modify the model parameters of the labeled data, and submit the training after the modification is completed. yolo detect mode = train data = datasets / ccmyxihu.yaml model = datasets / yolov8s.pt epochs = 300imgsz = 1280device = 2.

[0040] 7) After the training is completed, put the initially trained model in the folder

[0041] D:\cc\yolov8\ultralytics-main\ultralytics\cfg\runs\detect\train10\weights, and then copy the files to the model migration directory.

[0042] 8) In the model migration directory, run the command conda activate yolov8 to generate the type file.

[0043] 9) After preparing the relevant migration files, run the command pythonconvert.py .. / model / huangpujiang.onnx rk3588 to migrate the model trained by 4090GPU to RK3588 chip.

[0044] 10) Compile and export through the command exportGCC_COMPILER = / home / luxubt / yolov8 / host / bin / aarch64-buildroot-linux-gnu. / build-linux.sh -t rk3588 -a aarch64 -d yolov8.

[0045] 11) Finally, run the command to package the model into the new version that needs to be trained:

[0046] / home / luxubt / yolov8 / rknn_model_zoo / install / rk3588_linux_aarch64 / rknn_yolo v8_demo.

[0047] It should be noted that the algorithm used in the method of the present invention is YOLOv8, which is the latest version of the YOLO series. It introduces a series of innovations and improvements in the field of target detection to improve the speed and accuracy of detection. The following are some key features and innovations of the YOLOv8 algorithm:

[0048] 1. Brand-new SOTA model: YOLOv8 provides object detection networks with resolutions of P5640 and P61280, as well as instance segmentation models based on YOLACT. It provides models of different scales (N / S / M / L / X) to adapt to different application scenarios.

[0049] 2. Advanced Backbone and Neck Architecture: YOLOv8 uses an advanced convolutional neural network (CNN) as a backbone to extract multi-scale features from input images. This backbone may be an advanced version of CSPDarknet or other efficient architectures, which captures hierarchical feature maps representing low-level texture and high-level semantic information. In addition, the Neck module of YOLOv8 refines and fuses multi-scale features through an optimized Path Aggregation Network (PANet), which is crucial for detecting objects of different sizes and scales.

[0050] 3. Anchor-free Head: YOLOv8 has made significant improvements in the Head part, adopting the current mainstream decoupled head structure, separating the classification and detection heads, and transforming from an anchor-based method to an anchor-free method, simplifying the prediction process, reducing the number of hyperparameters, and improving the model's adaptability to objects of different aspect ratios and scales.

[0051] 4. Optimized loss function: YOLOv8 adopts the TaskAlignedAssigner positive sample assignment strategy in the calculation of loss function and introduces Distribution Focal Loss, which helps to improve the performance of the model in target detection tasks.

[0052] 5. Training method: YOLOv8's training method includes advanced data enhancement strategies, such as improved mosaic enhancement and mixup, which combine multiple images into one training sample, improving the model's generalization ability for different object scales, orientations, and spatial configurations. In addition, YOLOv8 also uses the focal loss function to deal with the problem of class imbalance in classification tasks.

[0053] 6. Reasoning process: The reasoning process of YOLOv8 is similar to that of YOLOv5, but it is necessary to decode the integral representation bbox in the Distribution FocalLoss and convert it into a regular 4-dimensional bbox format. Then, through a series of dimensional transformations, decoding, threshold filtering, and non-maximum suppression (NMS), the detection results are finally output.

[0054] Through these innovations, YOLOv8 provides an object detection tool that strikes a balance between speed and accuracy, suitable for a variety of application scenarios such as real-time video surveillance and autonomous vehicles.

[0055] In addition, the present invention also converts the trained YOLOv8 model into RKNN (Rockchip Neural Network), which is a model format designed specifically for the Rockchip NPU platform and ends with the .rknn suffix. It allows users to convert their self-developed algorithm models into RKNN models so that they can run on the Rockchip NPU platform.

[0056] Through the above steps, the complete process from the smart viewing device to model sample update, sample labeling, training, migration, and model update can be realized. In addition, the smart viewing device can also access the model training server through its own 4G network to update the latest model, thereby updating the AI ​​recognition capability of the smart viewing device.

[0057] S3, AI mapping: Based on the final recognition result, AI recognition is used to automatically configure the photo with illustrated explanation content, as well as the LOGO and digital photo frame to generate an electronic postcard.

[0058] After the AI ​​scene recognition model is updated, the scene recognition AI model is started. When the user takes a photo, the AI ​​scene recognition model will recognize the picture taken, and the "label name, confidence, and location coordinates" information can be obtained for each recognized object.

[0059] The specific process of step S3 is: by selecting the tag name of the object, the description graphic information of the scene corresponding to the tag name is filtered out from the device storage. The present invention adopts a mapping position selection algorithm in this step. According to the location of the scene, a reasonable position is selected from the left and right sides to place the scene description graphic information, so that the description is beautiful and easy to read, and does not block the scene. Specifically, the mapping position selection algorithm is: by default, the scene description graphic information is placed on the left side of the picture. If the calculated scene range coordinates are biased to the left side of the picture, the scene description graphic information is pasted to the right side of the picture.

[0060] For example, the display screen developed by the present invention has a resolution of 1920×1080px. After testing, a fixed scene illustration size of 300×360px was selected. At the beginning, the picture was fixedly placed at the left side of the photo (200px, 200px). Figure 3 As shown. However, it was later discovered that if some scenes are biased to the left when the user takes a photo, they will be blocked by the explanatory picture. Therefore, the present invention changes to dynamic calculation. First, the position of the scene recognition result box is calculated, and its x-coordinate range is 200px-400px. Then, the distance between the left boundary of the scene and the left boundary of the picture is calculated as 200px-0=200px, and the distance between the right boundary of the scene and the right boundary of the picture is 1920-400px=1520px. A larger distance is selected as the right distance. If the scene picture is biased to the left, then the explanatory picture is placed on the side with a larger distance - that is, the right side. Finally, through this method, in this example, the explanatory picture can be automatically placed in the blank position on the right side of the picture, such as (1200,1500), and the effect is beautiful.

[0061] Then paste the scenic spot logo on the lower left corner of the whole picture, paste the current date on the lower right corner, add a digital photo frame on the outermost side, call the image synthesis function in the ffmpeg image processing algorithm library, merge all elements by level, and finally generate an electronic postcard, such as Figure 4 shown.

[0062] In summary, according to the method of the present invention, tourists can adjust the zoom operation to a suitable viewing angle on the smart viewing device and take photos. The name of the core scenic spot will be automatically marked on the device screen, and accompanied by graphic annotations, scenic spot name, time and date, photo frame and other elements, to generate a beautiful photo that can be downloaded to the mobile phone. In addition, the device will also provide voice explanations based on the explanation materials prepared for the scene, so that tourists can understand the story behind the scene.

[0063] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating AI scene recognition photos based on intelligent scene viewing equipment, characterized in that: The following steps are involved: S1, tourists use smart viewing devices to obtain photos; S2. Use the AI ​​scene recognition model to identify scenic spots in the photos taken by tourists, including the tag name, confidence level and range coordinates of the scene in the picture; when the AI ​​scene recognition model recognizes multiple scenes, use the C-position analysis algorithm to locate a scene in the most central position as the final result of this recognition; S3. Based on the final recognition result, AI recognition is used to automatically configure the photo with illustrated explanation content, LOGO and digital photo frame to generate an electronic postcard; The intelligent viewing device at least includes a camera for image acquisition and a display screen capable of displaying the acquired images for tourists to browse. The device also has a core circuit board composed of a CPU, GPU or NPU processing chip to perform camera data processing.

2. The AI ​​scene recognition photo generation method based on the intelligent scene viewing device according to claim 1 is characterized in that: The specific process of step S1 is: The smart viewing device uses a camera as the image acquisition end and a display screen as the image output end. The human-computer interaction includes four buttons: zoom in, zoom out, take a photo, and AI function on / off. Tourists use the zoom in or zoom out button to adjust the image to a satisfactory optimal image position, and then click the photo button. At this time, the photo action triggers an AI recognition task, and at the same time, a background photo is obtained from the camera's video stream through a screenshot method, completing the tourist photo process.

3. The AI ​​scene recognition photo generation method based on the intelligent scene viewing device according to claim 1 is characterized in that: The AI ​​scene recognition model in step S2 is a scene recognition model obtained by training at least 100 photos of each scene based on a deep learning algorithm, and the deep learning algorithm adopts the YOLOv8 algorithm.

4. The AI ​​scene recognition photo generation method based on the intelligent scene viewing device according to claim 1 is characterized in that: The C position analysis algorithm in step S2 is: compare the center point Xn coordinates of all recognized objects with the center point Xo coordinates of the screen resolution, and select the object with the smallest distance to the center point Xo coordinates of the screen resolution as the recognized scene.

5. The AI ​​scene recognition photo generation method based on intelligent scene viewing equipment according to claim 1 is characterized in that: The specific process of step S3 is: by selecting the tag name of the object, the description graphic information of the scene corresponding to the tag name is filtered out from the device storage, and a reasonable position is selected from the left and right sides to place the scene description graphic information according to the location of the scene using a map position selection algorithm; at the same time, the scenic spot LOGO is pasted on the lower left corner of the entire picture, the current date is pasted on the lower right corner, a digital photo frame is added on the outermost side, and the image synthesis function in the ffmpeg image processing algorithm library is called to merge all elements hierarchically to finally generate an electronic postcard.

6. The AI ​​scene recognition photo generation method based on the intelligent scene viewing device according to claim 5 is characterized in that: The image posting position selection algorithm is as follows: by default, the scene description graphic information is placed on the left side of the image; if the calculated scene range coordinates are biased to the left side of the image, the scene description graphic information is posted on the right side of the image.

Images