Vehicle exterior environment shooting method and device, electronic equipment and storage medium

Abstract

The invention provides a vehicle exterior environment shooting method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence such as computer vision, intelligent driving and deep learning, and the method can comprise the steps: obtaining a target picture which is a vehicle exterior environment picture collected by a vehicle-mounted camera in a driving process; performing object recognition on the target picture to obtain the category and position information of a target object in the target picture; determining a quality evaluation result of the target picture according to the category and the position information of the target object; and in response to the fact that a shooting triggering condition is determined according to a quality evaluation result, a vehicle-mounted camera is used for shooting a media file aiming at the environment outside the vehicle, and the media file is a photo or a video.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to methods, devices, electronic devices and storage media for photographing an environment outside a vehicle in the fields of computer vision, intelligent driving and deep learning. Background Art

[0002] With the development of technology, current vehicles not only provide basic driving and travel functions, but also provide more and more other functions, such as the in-vehicle smart shooting function. Summary of the invention

[0003] The present invention provides a method, a device, an electronic device and a storage medium for photographing an environment outside a vehicle.

[0004] A method for photographing an environment outside a vehicle, comprising:

[0005] Acquire a target image, wherein the target image is an image of the vehicle's exterior environment captured by a vehicle-mounted camera during driving;

[0006] Performing object recognition on the target image to obtain the category and position information of the target object in the target image;

[0007] Determining a quality assessment result of the target image according to the category and position information of the target object;

[0008] In response to determining that a shooting trigger condition is met according to the quality assessment result, a media file is shot using the vehicle-mounted camera for the external vehicle environment, where the media file is a photo or a video.

[0009] A device for photographing an environment outside a vehicle, comprising: an acquisition module, a recognition module, an evaluation module and a photographing module;

[0010] The acquisition module is used to acquire a target image, wherein the target image is an image of the vehicle's external environment captured by the vehicle's camera during driving;

[0011] The recognition module is used to perform object recognition on the target image to obtain the category and position information of the target object in the target image;

[0012] The evaluation module is used to determine the quality evaluation result of the target image according to the category and position information of the target object;

[0013] The shooting module is used to respond to determining that the shooting trigger condition is met according to the quality assessment result, and use the vehicle-mounted camera to shoot a media file for the external vehicle environment, where the media file is a photo or a video.

[0014] An electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0018] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0019] A computer program product comprises a computer program / instruction, wherein the computer program / instruction implements the method described above when executed by a processor.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 This is a flow chart of a first embodiment of the method for photographing the vehicle's exterior environment disclosed in the present invention;

[0023] Figure 2 A schematic diagram of the media file and the corresponding content description information described in the present disclosure;

[0024] Figure 3 This is a flow chart of a second embodiment of the method for photographing the vehicle's exterior environment disclosed in the present invention;

[0025] Figure 4 Schematic diagram of the composition structure of the vehicle exterior environment shooting device embodiment 400 of the present disclosure;

[0026] Figure 5 A schematic block diagram of an electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] In addition, it should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0029] Figure 1 FIG. 1 is a flow chart of the first embodiment of the method for photographing the vehicle exterior environment disclosed in the present invention. Figure 1 As shown, the following specific implementation methods are included.

[0030] In step 101, a target image is acquired, where the target image is an image of the vehicle's exterior environment captured by a vehicle-mounted camera during driving.

[0031] In step 102, object recognition is performed on the target image to obtain the category and position information of the target object in the target image.

[0032] In step 103, a quality assessment result of the target image is determined according to the category and position information of the target object.

[0033] In step 104, in response to determining that the shooting trigger condition is met according to the quality assessment result, a media file is shot using a vehicle-mounted camera for the vehicle exterior environment, and the media file is a photo or a video.

[0034] In the traditional in-car smart shooting solution, users (such as car owners) can use the driving recorder to take photos and videos. For example, a snapshot button can be set on the steering wheel. When the user sees the beautiful scenery outside the car, he can long press the snapshot button to use the driving recorder to take a video and save it, or short press the snapshot button to use the driving recorder to take a photo and save it. Although this method can meet the user's needs for shooting the environment outside the car, it requires the user to operate manually, which will not only affect the user's driving process and thus affect the driving safety, but also when the vehicle speed is fast, it is easy to miss the shooting opportunity, thereby affecting the quality of the photos or videos taken.

[0035] By adopting the scheme described in the above method embodiment, the vehicle-mounted camera can be used to capture the image of the external environment of the vehicle, that is, to obtain the target image, and the quality assessment result of the target image can be determined by performing object recognition on the target image, and then the vehicle-mounted camera can be used to take photos or videos of the external environment when it is determined that the shooting trigger conditions are met according to the quality assessment results. The whole process can be completed automatically, which will not affect the user's driving process, thereby improving driving safety. Moreover, the shooting is performed when it is determined that the shooting trigger conditions are met according to the quality assessment results, thereby avoiding missing good shooting opportunities as much as possible and improving the quality of the photos or videos taken.

[0036] The specific location of the vehicle-mounted camera is not limited, for example, it can be located at the front side of the vehicle. The vehicle-mounted camera can be used to collect the external environment picture. For the convenience of description, the external environment picture can be called the target picture.

[0037] For the target screen, object recognition may be directly performed on it. Alternatively, in some embodiments of the present disclosure, in response to determining that the target screen meets the recognition trigger condition, object recognition may be performed on the target screen.

[0038] That is, after acquiring the target picture, it can be first determined whether it meets the recognition trigger condition. If so, object recognition can be performed on the target picture. Otherwise, the target picture can be discarded.

[0039] In some embodiments of the present disclosure, the target picture may be acquired periodically. Accordingly, in response to determining that the similarity between the target picture acquired this time and the target picture acquired most recently is less than a first threshold, it may be determined that the recognition trigger condition is met.

[0040] The time interval between two consecutive acquisitions of the target screen and the specific value of the first threshold can be determined according to actual needs. In addition, if it is determined that the target screen acquired this time has a large similarity with the target screen acquired most recently, it can be determined that the recognition trigger condition is not met, and the target screen acquired this time can be discarded. On the contrary, if the similarity is small, that is, less than the first threshold, it can be determined that the recognition trigger condition is met, and the target screen acquired this time can be recognized.

[0041] There is no restriction on how to measure the similarity between two pictures. For example, the mean square error (MSE) of the two pictures can be obtained. The smaller the mean square error, the greater the similarity between the two pictures. For another example, the feature vectors of the two pictures can be obtained respectively, and the similarity between the two feature vectors can be calculated, and the similarity between the two feature vectors can be used as the similarity between the two pictures. For another example, the histograms of the two pictures can be obtained respectively, and the similarity between the two histograms can be compared, and the similarity between the two histograms can be used as the similarity between the two pictures.

[0042] Through the above processing, frequent recognition and shooting of the same or similar pictures can be avoided, thereby reducing resource consumption, improving processing efficiency, and minimizing duplication of shooting results.

[0043] By performing object recognition on the target image, the category and location information of the target object can be obtained. Object recognition refers to the location and classification of objects in the image. The Region-Convolutional Neural Network (R-CNN) is designed for object recognition. It can identify the category and location information of objects in the image through five steps: candidate region generation, regional feature extraction, feature vector classification, bounding box regression, and recognition result output. However, since R-CNN runs CNN for each candidate region separately, the computational efficiency is low, resulting in a slow recognition speed.

[0044] Accordingly, the solution described in the present disclosure proposes an improved convolutional neural network architecture, namely, the Faster Regional Convolutional Neural Network (Faster R-CNN), which uses a Region Proposal Network (RPN) to generate candidate regions based on R-CNN.

[0045] Specifically, in some embodiments of the present disclosure, a feature map corresponding to the target screen can be first generated by CNN, and then a group of candidate areas of predetermined size can be generated based on the feature map, and the maximum probability value and fine bounding box information corresponding to each candidate area can be determined respectively, the maximum probability value is the probability value with the largest value selected after determining the probability values ​​that the objects in the candidate areas belong to different predetermined categories respectively, and then the target area can be determined based on the candidate areas, and the objects in the target area can be determined as target objects. Furthermore, for each target object, the following processing can be performed respectively: the category corresponding to the maximum probability value of the target area where the target object is located is determined as the category of the target object, and the fine bounding box information of the target area where the target object is located is determined as the position information of the target object.

[0046] Among them, the visual geometry group network 16 (VGG16, Visual Geometry Group Network16) can be used to extract the spatial features of the target image and generate a feature map, and the size of the feature map can be determined according to actual needs.

[0047] Then, a set of candidate regions of predetermined size may be generated according to the feature map. In some embodiments of the present disclosure, RPN may be used to generate the following: a set of candidate regions of predetermined size, initial bounding box information of each candidate region, and probability information of the existence of an object in each candidate region.

[0048] That is, RPN can be used to generate a set of candidate regions of a predetermined size by sliding a window on the feature map, and can generate initial bounding box information for each candidate region and probability information of the existence of an object in each candidate region. To distinguish it from other bounding box information, the bounding box information generated here is called initial bounding box information, and the bounding box information refers to the bounding box coordinate information in the target image. In addition, the probability information refers to the probability information of whether the candidate region contains the target to be identified.

[0049] For each candidate region, the corresponding maximum probability value and fine bounding box information can also be determined respectively. In some embodiments of the present disclosure, the following processing can be performed for each candidate region: the candidate region is mapped to a feature representation of a predetermined size through a region of interest (ROI) pooling algorithm, and the maximum probability value and fine bounding box information of the candidate region are generated using a fully connected layer based on the feature representation, the initial bounding box information and probability information of the candidate region, and the fine bounding box information is the bounding box information obtained after optimizing the initial bounding box information.

[0050] That is, for each candidate area, it can also be mapped to a feature representation of a predetermined size through the ROI pooling algorithm to ensure that the size of each candidate area is consistent, which is convenient for the fully connected layer to process. After that, target classification and bounding box regression can be performed, that is, the fully connected layer can be used to classify each candidate area (such as people, sunsets, lakes, trees, etc.), and regress to obtain more accurate bounding box information (fine bounding box information). Accordingly, for each candidate area, the feature representation, initial bounding box information and probability information corresponding to the candidate area can be used as the input of the fully connected layer to obtain the output maximum probability value and fine bounding box information. The maximum probability value is the probability value with the largest value selected from the probability values ​​of the objects in the candidate area belonging to different predetermined categories.

[0051] Furthermore, a target area can be determined based on each candidate area, and an object in the target area can be determined as a target object. Accordingly, for each target object, the category corresponding to the maximum probability value of the target area where the target object is located can be determined as the category of the target object, and the fine bounding box information of the target area where the target object is located can be determined as the location information of the target object.

[0052] The specific values ​​of the above-mentioned predetermined sizes can be determined according to actual needs. In addition, the above-mentioned processing method can improve the object recognition speed, such as achieving real-time object recognition, so as to better meet the needs of the vehicle-mounted scene described in the present disclosure.

[0053] In some embodiments of the present disclosure, a method of determining a target region based on each candidate region may include: performing redundant filtering processing on each candidate region using a non-maximum suppression (NMS) algorithm, and determining the remaining candidate regions as the target region.

[0054] The NMS algorithm can be used to remove redundant candidate regions, ensuring that only the best candidate regions are retained, and preventing multiple candidate regions from marking the same object (corresponding to the object in the solution described in the present invention). That is, the best candidate region (best box) representing the actual object can be accurately screened out, while removing excessive overlapping boxes to improve subsequent processing efficiency and accuracy.

[0055] After respectively acquiring the category and position information of each target object in the target picture, the quality assessment result of the target picture may be determined according to the category and position information of each target object.

[0056] In some embodiments of the present disclosure, the quality assessment results may include: a quality score, and a method for determining the quality score of a target image based on the category and position information of each target object may include: performing the following processing for each target object: determining the category score and the position score of the target object based on the category and position information of the target object, determining the object score of the target object in combination with the category score and the position score, and determining the quality score of the target image in combination with the object scores of each target object in the target image.

[0057] For example, corresponding scores can be set in advance for different categories such as people, flowers, sunsets, lakes, trees, and grasslands. Accordingly, the score corresponding to the category of the target object can be determined as the category score of the target object. Alternatively, different categories can be divided into different levels, and corresponding scores can be set for different levels. Each level can include at least one category. Accordingly, the score corresponding to the level to which the category of the target object belongs can be determined as the category score of the target object.

[0058] In addition, when determining the location score of the target object, the distance between the target object and the vehicle-mounted camera can be determined first based on the location information, and then the location score of the target object can be determined based on the distance and the popularity of the target object. For example, assuming that the target object is a building, the closer the distance between the building and the vehicle-mounted camera, the higher the location score can be. At the same time, the higher the popularity of the building, the higher the location score can be.

[0059] In some embodiments of the present disclosure, after respectively obtaining the category score and the position score of the target object, the category score and the position score may be multiplied by the corresponding weights, and the two products obtained may be added, and the sum of the additions may be determined as the object score of the target object. Furthermore, the average of the object scores of the target objects in the target picture may be obtained, and the average may be determined as the quality score of the target picture, or the maximum value of the object scores of the target objects in the target picture may be determined as the quality score of the target picture. The specific value of each weight may be determined according to actual needs.

[0060] For example, assuming that the target picture includes three target objects, namely target object 1, target object 2 and target object 3, and the corresponding object scores are object score 1, object score 2 and object score 3, then the average of object score 1, object score 2 and object score 3 can be calculated, and the average can be determined as the quality score of the target picture, or the maximum value can be selected from object score 1, object score 2 and object score 3, assumed to be object score 2, and then object score 2 can be determined as the quality score of the target picture.

[0061] Through the above processing, the object score of the target object can be determined in combination with the category and position information of the target object, and the quality score of the target picture can be determined in combination with the object scores of each target object in the target picture, thereby improving the accuracy of the acquired object score and quality score.

[0062] Further, whether the shooting trigger condition is met can be determined based on the quality score of the target image. In some embodiments of the present disclosure, in response to determining that the quality score of the target image is greater than a second threshold, it can be determined that the shooting trigger condition is met. The specific value of the second threshold can be determined according to actual needs.

[0063] If it is determined that the quality score of the target image is greater than the second threshold, then it can be determined that the shooting trigger conditions are met. Accordingly, for the outside vehicle environment, the vehicle-mounted camera can be used to take photos or videos, wherein the user can pre-set the shooting type, that is, set whether to take photos or videos when the shooting trigger conditions are met. Conversely, if it is determined that the quality score of the target image is less than or equal to the second threshold, it can be determined that the shooting trigger conditions are not met. Accordingly, the vehicle-mounted camera will not be used to take photos or videos, so as to avoid taking low-quality photos or videos and reduce resource consumption.

[0064] In actual applications, after determining that the shooting trigger conditions are met, the user can be asked whether to take the photo through voice or other means. If the user agrees to take the photo, the shooting command can be issued through voice or clicking a predetermined button, and then the on-board camera can be used to take photos or videos, thereby improving user participation.

[0065] In addition, in some embodiments of the present disclosure, when using a vehicle-mounted camera to shoot media files, the optimal shooting angle of the vehicle-mounted camera can be first determined based on the category and position information of the target object in the target screen, and then the shooting angle of the vehicle-mounted camera can be adjusted to the optimal shooting angle, and then the adjusted vehicle-mounted camera can be used to shoot media files.

[0066] There is no restriction on how to determine the best shooting angle of the vehicle-mounted camera. For example, the category and position information of the target object in the target image can be input into a pre-trained evaluation model to obtain the best shooting angle output by the evaluation model.

[0067] By adjusting the shooting angle of the vehicle camera to the optimal shooting angle, the quality of the captured photos and videos can be improved.

[0068] In some embodiments of the present disclosure, after shooting media files using a vehicle-mounted camera, storage configuration information can also be obtained, and the storage location of the media files can be determined based on the storage configuration information. The storage location may include: a vehicle storage space or a predetermined non-vehicle storage space, and the media files can then be stored in the determined storage location.

[0069] That is to say, the user can set the storage location of the captured media files by himself, which is very flexible and convenient. Preferably, considering the limited storage space of the vehicle computer, the media files can be stored in a predetermined non-vehicle storage space, such as the user's network disk.

[0070] In addition, in some embodiments of the present disclosure, when a media file is stored in a determined storage location, content description information corresponding to the media file may be generated, and the media file and the content description information may be stored together in the storage location.

[0071] There is no limitation on how to generate the content description information corresponding to the media file. For example, the media file can be used as the input of a pre-trained content generation model to obtain the output content description information.

[0072] Figure 2 Schematic diagram of the media file and the corresponding content description information described in the present disclosure. Figure 2 As shown, assuming that the media file is a photo, the corresponding content description information can be: This is a photo taken on ** month ** day of ** year. This photo gives people a sense of tranquility and peace. The setting sun, golden light sprinkled on the water, creating a warm and comfortable atmosphere. The boats and rowboats on the lake add a leisurely element, as if people are enjoying the good times of the evening. The shadows of trees and the outline of the city in the distance outline a harmonious natural and cultural landscape...

[0073] Through the above processing, users can view photos more conveniently and have a better understanding of the content of photos.

[0074] Combined with the above introduction, Figure 3 FIG. 2 is a flow chart of a second embodiment of the method for photographing the vehicle exterior environment disclosed in the present invention. Figure 3 As shown, the following specific implementation methods are included.

[0075] In step 301, a target image is acquired, where the target image is an image of the vehicle's exterior environment captured by a vehicle-mounted camera during driving.

[0076] In step 302, it is determined whether the target image meets the identification triggering condition. If so, step 303 is executed; otherwise, the process ends.

[0077] For example, the target screen may be acquired periodically. In response to determining that the similarity between the target screen acquired this time and the target screen acquired most recently is less than a first threshold, it may be determined that the recognition trigger condition is met. Otherwise, it may be determined that the recognition trigger condition is not met and the process may be terminated.

[0078] In step 303, Faster R-CNN is used to perform object recognition on the target image to obtain the category and position information of the target object in the target image.

[0079] In step 304, for each target object, a category score and a location score of the target object are determined according to the category and location information of the target object, and an object score of the target object is determined in combination with the category score and the location score.

[0080] For example, the category score and the position score may be multiplied by the corresponding weights respectively, and the two products obtained may be added together, and the sum of the additions may be determined as the object score.

[0081] In step 305, a quality score of the target picture is determined in combination with the object score of the target object in the target picture.

[0082] For example, an average of the object scores of the target objects in the target picture may be obtained and determined as the quality score, or a maximum value of the object scores of the target objects in the target picture may be determined as the quality score.

[0083] In step 306, it is determined whether the quality score is greater than a second threshold, if so, step 307 is executed, otherwise, the process ends.

[0084] In step 307, the best shooting angle of the vehicle camera is determined according to the category and position information of the target object in the target image, and the shooting angle of the vehicle camera is adjusted to the best shooting angle, and the adjusted vehicle camera is used to shoot media files, which are photos or videos.

[0085] In step 308, content description information corresponding to the media file is generated.

[0086] In step 309 , the media file and the corresponding content description information are stored in a predetermined non-vehicle storage space, and then the process ends.

[0087] It is assumed in this embodiment that media files and the like are stored in a predetermined non-vehicle storage space, such as a user's network disk.

[0088] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure. In addition, for parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0089] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0090] Figure 4 FIG. 4 is a schematic diagram of the structure of the vehicle exterior environment photographing device embodiment 400 described in the present disclosure. Figure 4 As shown, it includes: an acquisition module 401, a recognition module 402, an evaluation module 403 and a shooting module 404.

[0091] The acquisition module 401 is used to acquire a target image, where the target image is an image of the vehicle's exterior environment captured by the vehicle's onboard camera during driving.

[0092] The recognition module 402 is used to perform object recognition on the target picture to obtain the category and position information of the target object in the target picture.

[0093] The evaluation module 403 is used to determine the quality evaluation result of the target image according to the category and position information of the target object.

[0094] The shooting module 404 is used to shoot media files using a vehicle-mounted camera for the vehicle exterior environment in response to determining that the shooting trigger condition is met according to the quality assessment result. The media files are photos or videos.

[0095] By adopting the scheme described in the above-mentioned device embodiment, the vehicle-mounted camera can be used to capture the image of the environment outside the vehicle, that is, to obtain the target image, and the quality assessment result of the target image can be determined by performing object recognition on the target image, and then the vehicle-mounted camera can be used to take photos or videos of the environment outside the vehicle when it is determined that the shooting trigger conditions are met according to the quality assessment results. The whole process can be completed automatically, so as not to affect the user's driving process, thereby improving driving safety. Moreover, the shooting is performed when it is determined that the shooting trigger conditions are met according to the quality assessment results, thereby avoiding missing good shooting opportunities as much as possible and improving the quality of the photos or videos taken.

[0096] For the target picture acquired by the acquisition module 401 , the recognition module 402 may directly perform object recognition on it, or, in some embodiments of the present disclosure, the recognition module 402 may perform object recognition on the target picture in response to determining that the target picture meets the recognition trigger condition.

[0097] In some embodiments of the present disclosure, the acquisition module 401 may periodically acquire the target picture. Accordingly, the recognition module 402 may determine that the recognition trigger condition is met in response to determining that the similarity between the target picture acquired this time and the target picture acquired most recently is less than a first threshold.

[0098] Furthermore, the recognition module 402 may perform object recognition on the target picture to obtain the category and position information of the target object in the target picture.

[0099] In some embodiments of the present disclosure, the recognition module 402 may first generate a feature map corresponding to the target screen through CNN, and then generate a set of candidate areas of predetermined size based on the feature map, and respectively determine the maximum probability value and fine bounding box information corresponding to each candidate area, wherein the maximum probability value is the probability value with the largest value selected from the probability values ​​of the objects in the candidate areas respectively belonging to different predetermined categories, and then the target area may be determined based on the candidate areas, and the objects in the target area may be determined as target objects, and the following processing may be performed on each target object: the category corresponding to the maximum probability value of the target area where the target object is located is determined as the category of the target object, and the fine bounding box information of the target area where the target object is located is determined as the position information of the target object.

[0100] In some embodiments of the present disclosure, the recognition module 402 may generate the following content using RPN based on the feature map: a set of candidate regions of a predetermined size, initial bounding box information of each candidate region, and probability information of the existence of an object in each candidate region.

[0101] In some embodiments of the present disclosure, the recognition module 402 may also perform the following processing for each candidate area: map the candidate area to a feature representation of a predetermined size through an ROI pooling algorithm, and generate a maximum probability value and fine bounding box information of the candidate area using a fully connected layer based on the feature representation, the initial bounding box information and probability information of the candidate area, where the fine bounding box information is the bounding box information obtained after optimizing the initial bounding box information.

[0102] In addition, in some embodiments of the present disclosure, the identification module 402 may determine the target area according to the candidate areas by using an NMS algorithm to perform redundancy filtering on the candidate areas and determine the remaining candidate areas as the target area.

[0103] After respectively acquiring the category and position information of each target object in the target picture, the evaluation module 403 may determine the quality evaluation result of the target picture according to the category and position information of each target object.

[0104] In some embodiments of the present disclosure, the quality assessment results may include: a quality score, and the way in which the assessment module 403 determines the quality score of the target image based on the category and position information of each target object may include: performing the following processing for each target object: determining the category score and the position score of the target object based on the category and position information of the target object, determining the object score of the target object in combination with the category score and the position score, and determining the quality score of the target image in combination with the object scores of each target object in the target image.

[0105] In some embodiments of the present disclosure, after respectively obtaining the category score and the position score of the target object, the evaluation module 403 may respectively multiply the category score and the position score by the corresponding weights, and may add the two products obtained, and determine the sum as the object score of the target object. Furthermore, the mean of the object scores of the target objects in the target picture may be obtained, and the mean may be determined as the quality score of the target picture, or the maximum value of the object scores of the target objects in the target picture may be determined as the quality score of the target picture.

[0106] Further, the shooting module 404 may determine whether the shooting trigger condition is met according to the quality score of the target image. In some embodiments of the present disclosure, the shooting module 404 may determine that the shooting trigger condition is met in response to determining that the quality score of the target image is greater than a second threshold.

[0107] In addition, in some embodiments of the present disclosure, when the shooting module 404 uses a vehicle-mounted camera to shoot media files, it can first determine the optimal shooting angle of the vehicle-mounted camera based on the category and position information of the target object in the target picture, and then adjust the shooting angle of the vehicle-mounted camera to the optimal shooting angle, and then use the adjusted vehicle-mounted camera to shoot media files.

[0108] In some embodiments of the present disclosure, after using the vehicle-mounted camera to shoot the media files, the shooting module 404 can also obtain storage configuration information, and can determine the storage location of the media files based on the storage configuration information. The storage location may include: vehicle storage space or predetermined non-vehicle storage space, and the media files can be stored in the determined storage location.

[0109] Furthermore, in some embodiments of the present disclosure, when storing the media file in the determined storage location, the shooting module 404 may also generate content description information corresponding to the media file, and then store the media file and the content description information together in the storage location.

[0110] Figure 4 The specific working process of the illustrated device embodiment can refer to the relevant description in the aforementioned method embodiment and will not be described in detail.

[0111] In summary, by adopting the scheme described in the present invention, the external environment of the vehicle can be automatically identified during driving, and when it is determined that the shooting trigger conditions are met based on the identification results, the shooting angle of the vehicle camera can be automatically adjusted to take photos or videos, thereby reducing disturbance to users, improving driving safety, and improving the quality of photos or videos taken.

[0112] The solution disclosed in the present disclosure can be applied to the field of artificial intelligence, especially computer vision, intelligent driving, and deep learning. Artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It has both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0113] In addition, the target images, photos, and videos in the embodiments described in this disclosure are not targeted at a specific user and do not reflect the personal information of a specific user. In the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0114] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0115] Figure 5 A schematic block diagram of an electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0116] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0117] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0118] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI, Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP, Digital Signal Processing), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the methods described in the present disclosure. For example, in some embodiments, the methods described in the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the methods described in the present disclosure may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the method described in the present disclosure in any other appropriate manner (for example, by means of firmware).

[0119] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0121] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM, Electronically Programmable Read-Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM, Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0124] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0126] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for photographing an environment outside a vehicle, comprising: Acquire a target image, wherein the target image is an image of the vehicle's exterior environment captured by a vehicle-mounted camera during driving; Performing object recognition on the target image to obtain the category and position information of the target object in the target image; Determining a quality assessment result of the target image according to the category and position information of the target object; In response to determining that a shooting trigger condition is met according to the quality assessment result, a media file is shot using the vehicle-mounted camera for the external vehicle environment, where the media file is a photo or a video.

2. The method according to claim 1, wherein: The performing object recognition on the target picture includes: in response to determining that the target picture meets a recognition trigger condition, performing object recognition on the target picture.

3. The method according to claim 2, wherein: The acquiring the target picture includes: periodically acquiring the target picture; The responding to determining that the target picture meets the recognition trigger condition includes: responding to determining that the similarity between the target picture acquired this time and the target picture acquired most recently is less than a first threshold, determining that the recognition trigger condition is met.

4. The method according to claim 1, wherein: The performing object recognition on the target picture to obtain the category and position information of the target object in the target picture includes: Generate a feature map corresponding to the target image through a convolutional neural network; Generate a set of candidate regions of a predetermined size according to the feature map, and determine the maximum probability value and fine bounding box information corresponding to each candidate region, wherein the maximum probability value is the maximum probability value selected from the probability values ​​of the objects in the candidate region belonging to different predetermined categories respectively; Determine a target area according to each candidate area, and determine an object in the target area as the target object; For each target object, the following processing is performed respectively: the category corresponding to the maximum probability value of the target area where the target object is located is determined as the category of the target object, and the fine bounding box information of the target area where the target object is located is determined as the position information of the target object.

5. The method according to claim 4, wherein: Generating a set of candidate regions of predetermined size according to the feature map comprises: According to the feature map, a region proposal network is used to generate the following: a set of candidate regions of a predetermined size, initial bounding box information of each candidate region, and probability information of an object existing in each candidate region.

6. The method according to claim 5, wherein: The determining of the maximum probability value and the refined bounding box information corresponding to each candidate region includes: For each candidate area, the following processing is performed respectively: the candidate area is mapped into a feature representation of a predetermined size through a region of interest pooling algorithm, and the maximum probability value and the refined bounding box information of the candidate area are generated using a fully connected layer based on the feature representation, the initial bounding box information of the candidate area, and the probability information. The refined bounding box information is the bounding box information obtained after optimizing the initial bounding box information.

7. The method according to claim 4, wherein: Determining the target region according to the candidate regions includes: performing redundant filtering processing on the candidate regions using a non-maximum suppression algorithm, and determining the remaining candidate regions as the target region.

8. The method according to claim 1, wherein: The quality assessment results include: quality score; Determining the quality assessment result of the target image according to the category and position information of the target object includes: For each target object, the following processing is performed respectively: according to the category and position information of the target object, the category score and the position score of the target object are determined respectively, and the object score of the target object is determined by combining the category score and the position score; The quality score is determined in combination with the object score of the target object in the target picture.

9. The method according to any one of claims 1 to 8, wherein: The using the vehicle-mounted camera to shoot media files includes: Determining the best shooting angle of the vehicle-mounted camera according to the category and position information of the target object in the target picture; The shooting angle of the vehicle-mounted camera is adjusted to the optimal shooting angle, and the media file is shot using the adjusted vehicle-mounted camera.

10. The method according to any one of claims 1 to 8, further comprising: After the media file is photographed by the vehicle-mounted camera, storage configuration information is obtained, and a storage location of the media file is determined according to the storage configuration information. The storage location includes: a vehicle storage space or a predetermined non-vehicle storage space, and the media file is stored in the storage location.

11. The method according to claim 10, wherein: Storing the media file in the storage location includes: Generate content description information corresponding to the media file; The media file and the content description information are stored together in the storage location.

12. A device for photographing an environment outside a vehicle, comprising: Acquisition module, recognition module, evaluation module and shooting module; The acquisition module is used to acquire a target image, wherein the target image is an image of the vehicle's external environment captured by the vehicle's camera during driving; The recognition module is used to perform object recognition on the target image to obtain the category and position information of the target object in the target image; The evaluation module is used to determine the quality evaluation result of the target image according to the category and position information of the target object; The shooting module is used to respond to determining that the shooting trigger condition is met according to the quality assessment result, and use the vehicle-mounted camera to shoot a media file for the external vehicle environment, where the media file is a photo or a video.

13. The device according to claim 12, wherein: In response to determining that the target picture meets a recognition trigger condition, the recognition module performs object recognition on the target picture.

14. The device according to claim 13, wherein: The acquisition module periodically acquires the target picture; In response to determining that the similarity between the target picture acquired this time and the target picture acquired most recently is less than a first threshold, the recognition module determines that the recognition trigger condition is met.

15. The device according to claim 12, wherein: The recognition module generates a feature map corresponding to the target screen through a convolutional neural network, generates a set of candidate areas of predetermined size according to the feature map, and determines the maximum probability value and fine bounding box information corresponding to each candidate area, wherein the maximum probability value is the probability value with the largest value selected after determining the probability values ​​that the objects in the candidate areas belong to different predetermined categories, determines a target area according to each candidate area, and determines the object in the target area as the target object, and performs the following processing on each target object: determines the category corresponding to the maximum probability value of the target area where the target object is located as the category of the target object, and determines the fine bounding box information of the target area where the target object is located as the position information of the target object.

16. The device according to claim 15, wherein: The recognition module generates the following content using a region proposal network according to the feature map: a set of candidate regions of a predetermined size, initial bounding box information of each candidate region, and probability information of an object existing in each candidate region.

17. The device according to claim 16, wherein: The recognition module performs the following processing on each candidate area: maps the candidate area to a feature representation of a predetermined size through a region of interest pooling algorithm, and generates the maximum probability value and the refined bounding box information of the candidate area using a fully connected layer based on the feature representation, the initial bounding box information of the candidate area, and the probability information, wherein the refined bounding box information is the bounding box information obtained after optimizing the initial bounding box information.

18. The device according to claim 15, wherein: The recognition module uses a non-maximum suppression algorithm to perform redundant filtering on each candidate area, and determines the remaining candidate area as the target area.

19. The device according to claim 12, wherein: The quality assessment results include: quality score; The evaluation module performs the following processing for each target object: determines the category score and the location score of the target object according to the category and location information of the target object, determines the object score of the target object in combination with the category score and the location score, and determines the quality score in combination with the object score of the target object in the target image.

20. The device according to any one of claims 12 to 19, wherein: The shooting module determines the best shooting angle of the vehicle camera according to the category and position information of the target object in the target picture, adjusts the shooting angle of the vehicle camera to the best shooting angle, and uses the adjusted vehicle camera to shoot the media file.

21. The device according to any one of claims 12 to 19, wherein: The shooting module is further used to obtain storage configuration information after shooting the media file using the vehicle-mounted camera, and determine the storage location of the media file according to the storage configuration information, wherein the storage location includes: a vehicle storage space or a predetermined non-vehicle storage space, and the media file is stored in the storage location.

22. The device according to claim 21, wherein The shooting module is further used to generate content description information corresponding to the media file, and store the media file and the content description information together in the storage location.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 11.

25. A computer program product, comprising a computer program / instruction, which implements the method according to any one of claims 1 to 11 when executed by a processor.

Images