Scene recognition method, device, server, storage medium and program product

Abstract

The disclosed embodiments relate to a scene recognition method, device, server, storage medium and program product. The method comprises: obtaining environmental information during vehicle driving; the environmental information comprises multiple frames of images corresponding to multiple lighting scenes; obtaining scene recognition results corresponding to the environmental information; the scene recognition results are obtained by processing each frame of the multiple frames using different adjustment factors and a preset recognition model; and performing corresponding driving operations according to the scene recognition results. The method can improve the recognition accuracy of dynamic scenes.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of video technology, and in particular, to a scene recognition method, device, server, storage medium, and program product. Background Art

[0002] With the progress of urban social life and the improvement of urban infrastructure, the scenarios faced by vehicle perception in autonomous driving scenarios are becoming increasingly complex, and a variety of dynamic scenes often appear, such as pedestrians walking, running or chasing, vehicles overtaking, changing lanes or stopping suddenly, etc. At the same time, they also have to face different lighting change scenes such as day / night, sunny / cloudy / rainy days, etc. In traditional technologies, the recognition of dynamic scenes in autonomous driving mainly analyzes single-frame static images in dynamic scenes to identify dynamic scenes.

[0003] However, traditional dynamic scene recognition methods have the problem of low recognition accuracy. Summary of the invention

[0004] The embodiments of the present disclosure provide a scene recognition method, device, server, storage medium, and program product, which can be used to improve the recognition accuracy of dynamic scenes.

[0005] In a first aspect, an embodiment of the present disclosure provides a scene recognition method, the method comprising:

[0006] Acquire environmental information during vehicle driving; the environmental information includes multiple frames of images corresponding to multiple lighting scenes;

[0007] Obtaining a scene recognition result corresponding to the environmental information; the scene recognition result is obtained by processing each frame of the multiple images using different adjustment factors and a preset recognition model;

[0008] The corresponding driving operation is performed according to the scene recognition result.

[0009] In a second aspect, an embodiment of the present disclosure provides a scene recognition device, the device comprising:

[0010] A first acquisition module is used to acquire environmental information during the driving process of the vehicle; the environmental information includes multiple frames of images corresponding to multiple lighting scenes;

[0011] A second acquisition module is used to acquire a scene recognition result corresponding to the environmental information; the scene recognition result is obtained by processing each frame of the multiple images using different adjustment factors and a preset recognition model;

[0012] An execution module is used to execute corresponding driving operations according to the scene recognition result.

[0013] In a third aspect, an embodiment of the present disclosure provides a server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0014] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect above.

[0015] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0016] The scene recognition method, device, server, storage medium and program product provided by the embodiments of the present disclosure acquire environmental information during vehicle driving, including multiple frame images corresponding to multiple lighting scenes. By adopting different adjustment factors, the illumination of each frame image in the multiple frame images can be adjusted, and frame images under multiple lighting scenes can be adjusted and processed with different dynamic ranges, so as to mine scene features under different lighting conditions to multiple degrees, so that the preset recognition model can be used to accurately obtain the scene recognition result corresponding to the environmental information during vehicle driving, and then the corresponding driving operation can be accurately performed according to the obtained scene recognition result, thereby ensuring the accuracy of the executed driving operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1a A diagram showing an application environment of a scene recognition method in an embodiment;

[0018] Figure 1b A diagram showing an application environment of a scene recognition method in an embodiment;

[0019] Figure 2 is a schematic flow chart of a scene recognition method in an embodiment;

[0020] Figure 3 is a flow chart of a scene recognition method in another embodiment;

[0021] Figure 4 is a flow chart of a scene recognition method in another embodiment;

[0022] Figure 5 is a flow chart of a scene recognition method in another embodiment;

[0023] Figure 6 is a flow chart of a scene recognition method in another embodiment;

[0024] Figure 7 is a structural block diagram of a scene recognition device in one embodiment;

[0025] Figure 8 FIG. 4 is a diagram showing the internal structure of a server in one embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure more clear, the embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present disclosure and are not used to limit the embodiments of the present disclosure.

[0027] First, before specifically introducing the technical solution of the embodiment of the present disclosure, the technical background or technical evolution context on which the embodiment of the present disclosure is based is introduced. Generally, in the field of autonomous driving perception, the current technical background is: analyzing a single-frame static image in a dynamic scene to identify the dynamic scene, resulting in the problem of low recognition accuracy in the perception and recognition of the dynamic scene. Based on this background, the applicant has found through long-term model simulation research and development and the collection, demonstration and verification of experimental data that in a variety of dynamic scenes, such as pedestrians walking, running or chasing, vehicles overtaking, changing lanes or stopping suddenly, etc., at the same time, they also have to face different lighting change scenes such as day / night, sunny / cloudy / rainy days, etc., and the use of a single-frame static image analysis method to identify the above-mentioned dynamic scenes will result in a problem of low recognition accuracy. How to accurately identify dynamic scenes has become a problem that needs to be solved urgently. In addition, it should be noted that the applicant has made a lot of creative work from determining that the use of a single-frame static image analysis method to identify dynamic scenes has a low accuracy problem and the technical solution introduced in the following embodiments.

[0028] The technical solutions involved in the embodiments of the present disclosure are introduced below in combination with the scenarios to which the embodiments of the present disclosure are applied.

[0029] The scene recognition method provided by the embodiment of the present disclosure can be applied to Figure 1a The application environment shown or can be applied to Figure 1b In the application environment shown. Figure 1a In the application environment shown, the vehicle includes a video acquisition device and a central control device. The video acquisition device can be but is not limited to various cameras. The video acquisition device can acquire video frame images during the driving of the vehicle; the central control device of the vehicle can process the received data. For example, the video acquisition device can send the video frame images acquired during the driving of the vehicle to the central control device. The central control device can process the received video frame images, identify the scenes corresponding to the received video frame images, and so on.

[0030] For Figure 1bIn the application environment shown, a vehicle 102 communicates with a server 104 via a network. The vehicle 102 may include a video acquisition device, which may be, but is not limited to, various cameras, etc. The server 104 may send an inquiry instruction to the vehicle, and the user views the inquiry instruction and, with the authorization and consent of the user, the vehicle 102 uploads the video frame image collected by the video acquisition device to the server 104, or, after the vehicle and the server establish a communication connection, the video frame image collected by the acquisition device may be uploaded to the server 104 in real time, and the server 104 may process the received video frame image. Optionally, the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0031] In one embodiment, Figure 2 As shown, a scene recognition method is provided, comprising the following steps:

[0032] S201, obtaining environmental information during vehicle driving; the environmental information includes multiple frames of images corresponding to multiple lighting scenes.

[0033] Wherein, the environmental information is a plurality of frames of images corresponding to a plurality of illumination scenes in a dynamic scene collected by a video acquisition device of the vehicle during the driving process of the vehicle, and the plurality of illumination scenes may include but are not limited to changes from day to night, from sunny to cloudy, from sunny to rainy, or from cloudy to rainy, or at least one scene of other illumination changes in entering and exiting a tunnel. Exemplarily, the environmental information may be a plurality of frames of images corresponding to changes from day to night on a certain road during the driving process of the vehicle, or may be a plurality of frames of images corresponding to changes from rainy to sunny on a certain road during the driving process of the vehicle, or the environmental information may include a plurality of frames of images corresponding to a plurality of scenes on a certain road during the driving process of the vehicle, for example, the environmental information may include a plurality of frames of images corresponding to changes from day to night on the road during the driving process of the vehicle, and a plurality of frames of images corresponding to changes from rainy to sunny on the road. Optionally, the video acquisition device may be a vehicle-mounted camera, that is, the environmental information may be collected by a vehicle-mounted camera, and it is understandable that the environmental information may be collected by one vehicle-mounted camera or by multiple vehicle-mounted cameras.

[0034] Optionally, after obtaining the above environmental information, the obtained environmental information can also be filtered to remove noise information in the environmental information. It can be understood that in this application, if the execution subject is Figure 1a The central control device of the vehicle shown in , in this embodiment, is the above-mentioned environmental information obtained by the central control device of the vehicle, and the central control device performs filtering processing on the obtained environmental information; in this application, if the execution subject is Figure 1bThe server shown in this embodiment is a server that receives environmental information sent by the vehicle during the vehicle's driving process and performs filtering processing on the acquired environmental information.

[0035] S202, obtaining a scene recognition result corresponding to the environmental information; the scene recognition result is obtained by processing each frame of the multiple frames of images using different adjustment factors and a preset recognition model.

[0036] It can be understood that in this embodiment, the scene recognition results corresponding to the above-mentioned environmental information can be obtained by the central control device on the vehicle, or by a server connected to the vehicle through a network. Furthermore, if the scene recognition results in this embodiment are obtained by a server connected to the vehicle through a network, the server also needs to send the obtained scene recognition results to the vehicle through the network.

[0037] It should be noted that the following embodiments are described using the example of an execution subject being a server connected to the vehicle via a network. When the execution subject is the central control device of the vehicle, the principles implemented by the central control device are the same as those implemented when the execution subject is a server. The following embodiments will no longer repeat the implementation principles when the execution subject is the central control device of the vehicle.

[0038] Optionally, the server may process each frame of the multiple images corresponding to the multiple illumination scenes included in the acquired environmental information using different adjustment factors and a preset recognition model to obtain a scene recognition result corresponding to the above environmental information. Optionally, the server may adjust the illumination of each frame of the multiple images using different adjustment factors, and input each adjusted frame of the image into the above recognition model to obtain a scene recognition result corresponding to the above environmental information. Alternatively, the server may also input each frame of the multiple images and different adjustment factors into the above recognition model, and the recognition model processes each frame of the multiple images using different adjustment factors, and then obtain a scene recognition result corresponding to the above environmental information. Among them, the scene recognition result corresponding to the above environmental information may be a recognition result of pedestrian information on the road during vehicle driving, or a recognition result of a vehicle traveling on the road during vehicle driving. For example, the scene recognition result may be a scene of pedestrians walking on the road, a scene of pedestrians running on the road, a scene of pedestrians chasing on the road, or a scene of vehicles overtaking on the road, or a scene of vehicles changing lanes on the road, or another example may be a scene of vehicles stopping suddenly on the road.

[0039] S203: Execute corresponding driving operations according to the scene recognition result.

[0040] Optionally, different scene recognition results may correspond to different driving operations, or may correspond to the same driving operation. For example, if the scene recognition result corresponding to the above environmental information is a scene of pedestrians walking on the road, the corresponding driving operation performed by the vehicle's central control device may be to maintain the current vehicle speed; if the scene recognition result corresponding to the above environmental information is a scene of pedestrians running on the road or pedestrians chasing on the road, the corresponding driving operation performed by the vehicle's central control device may be to slowly reduce the vehicle speed; if the scene recognition result corresponding to the above environmental information is a scene of a vehicle overtaking on the road, the corresponding driving operation performed by the vehicle's central control device may be any one of the driving operations of reducing the vehicle speed or maintaining driving in the current lane. If the scene recognition result corresponding to the above environmental information is a scene of a vehicle changing lanes on the road, the corresponding driving operation performed by the vehicle's central control device may be any one of the driving operations of reducing the vehicle speed or maintaining driving in the current lane. If the scene recognition result corresponding to the above environmental information is a scene of a vehicle stopping suddenly on the road, the corresponding driving operation performed by the vehicle's central control device may be an emergency braking operation.

[0041] In the above-mentioned scene recognition method, the environmental information obtained during the vehicle driving process includes multiple frames of images corresponding to multiple lighting scenes. Different adjustment factors can be used to adjust the lighting of each frame in the multiple frames, and frame images under multiple lighting scenes can be adjusted and processed with different dynamic ranges. Scene features under different lighting conditions can be mined to multiple degrees, so that the preset recognition model can be used to accurately obtain the scene recognition results corresponding to the environmental information during the vehicle driving process, and then the corresponding driving operations can be accurately performed according to the obtained scene recognition results, thereby ensuring the accuracy of the executed driving operations.

[0042] Further, based on the above embodiment, in one embodiment, Figure 3 As shown, the above S202 includes:

[0043] S301, using different adjustment factors to adjust the illumination in each frame of multiple frames to obtain a target frame image.

[0044] It can be understood that the adjustment factor in this embodiment is used to adjust the illumination in each frame image corresponding to multiple illumination scenes, and the adjustment factor is a factor related to illumination. Optionally, the server can use different adjustment factors to adjust the pixel value of each frame image in the above-mentioned multiple frames of images to adjust the illumination in each frame image in the multiple frames of images, or the server can use different adjustment factors to adjust the RGB value of each frame image in the above-mentioned multiple frames of images to adjust the illumination in each frame image in the video frame sequence. Optionally, the server can determine each adjusted frame image as a target frame image, or can perform fusion processing on each adjusted frame image and determine the frame image after fusion processing as the target frame image.

[0045] S302, inputting the target frame image into a preset recognition model to obtain a scene recognition result corresponding to the environmental information.

[0046] Optionally, the preset recognition model can be a recognition model of various network structures, for example, the recognition model can be a recognition model of any network structure among VGG, ResNet, and I3D. It is understandable that if the target frame image obtained by the server includes multiple frame images, the server can input the multiple frame images into the preset recognition model respectively, obtain the recognition results corresponding to each frame image, and then fuse the recognition results corresponding to each frame image to obtain the scene recognition result corresponding to the above environmental information; if the target frame image obtained by the server includes a frame image, the server can input the target frame image into the preset recognition model, and use the recognition result of the frame image obtained as the scene recognition result corresponding to the above environmental information. Exemplarily, if the collected multiple frame images are multiple frame images of pedestrians running on the road under multiple lighting scenes, the server can use different adjustment factors to adjust the lighting in each frame image of the multiple frame images to obtain the target frame image, input the target frame image into the above recognition model, and the scene recognition result corresponding to the environmental information obtained during the vehicle driving process is that pedestrians are running on the road.

[0047] In this embodiment, the server uses different adjustment factors to adjust the illumination in each frame of multiple frames, so that the frame images under various lighting scenes can be adjusted with different dynamic ranges, and the scene features under different lighting scenes can be mined to various degrees. In other words, the obtained target frame image includes scene features under various lighting scenes, and the obtained target frame image is input into a preset recognition model, which can accurately identify the scene corresponding to the environmental information during the vehicle driving process, thereby improving the accuracy of the obtained scene recognition result.

[0048] In the above scenario where different adjustment factors are used to adjust the illumination in each frame of multiple frames to obtain a target frame image, in one embodiment, Figure 4 As shown, the above S301 includes:

[0049] S401, using different adjustment factors to adjust the illumination in each frame of a plurality of frames of images, to obtain an adjusted frame image corresponding to each adjustment factor.

[0050] Optionally, in this embodiment, the number of different adjustment factors can be set according to the recognition requirements of the scene, and the adjustment factors in different lighting scenes are also different. Exemplarily, if the environmental information obtained is environmental information in an overexposure scene, the server needs to use the adjustment factor to adjust the illumination in each frame image in the environmental information to within a preset illumination range to obtain an adjusted frame image; if the environmental information obtained is environmental information in an underexposure scene, the server needs to use the adjustment factor to adjust the illumination in each frame image in the environmental information to within a preset illumination range to obtain an adjusted frame image. Optionally, the server can use different adjustment factors to obtain the product of the pixel value of each frame image in the above-mentioned multiple frames of images and the adjustment factor, and determine the product value of the pixel value of each frame image and the adjustment factor as the pixel value of the adjusted frame image.

[0051] S402: Fusing the adjusted frame images to obtain a target frame image.

[0052] Specifically, after the server obtains the adjusted frame images corresponding to each adjustment factor, it can determine the fused frame images as the target frame images by fusing the adjusted frame images. Optionally, the server can fuse the adjusted frame images according to the generation order of the adjusted frame images corresponding to the adjustment factors to obtain the above-mentioned target frame images. Taking the three frames of images corresponding to multiple lighting scenes included in the above-mentioned environmental information as an example, the server can use three different adjustment factors to adjust the illumination in the three frames of images respectively to obtain the adjusted frame images corresponding to the three different adjustment factors, and then fuse the adjusted frame images corresponding to the three different adjustment factors according to the generation order of the adjusted frame images corresponding to the three different adjustment factors to obtain the above-mentioned target frame images. Optionally, in this embodiment, the fusion method of the server fusing the adjusted frame images can be a stacking fusion method, or a fusion method of taking the average, or other fusion methods.

[0053] In this embodiment, the server uses different adjustment factors to adjust the illumination in each frame of multiple frames of environmental information obtained during the vehicle's driving process, so that the adjusted frame images corresponding to each adjustment factor can be accurately obtained, and then the adjusted frame images can be fused to accurately obtain the target frame image. In this way, the target frame image is input into a preset recognition model, and the scene recognition result corresponding to the environmental information can be accurately obtained through the recognition model, thereby improving the accuracy of the scene recognition result corresponding to the environmental information.

[0054] In the above-mentioned scenario where different adjustment factors are used to adjust the illumination in each frame of multiple images to obtain adjusted frame images corresponding to each adjustment factor, in one embodiment, the above-mentioned S401 includes: using different adjustment factors to adjust the RGB value of each frame of multiple images to obtain each adjusted frame image.

[0055] It can be understood that the RGB values ​​of the image correspond to the values ​​of the three channels. Accordingly, the server can use the adjustment factors of the three channels corresponding to the RGB values ​​to adjust the RGB values ​​of each frame of the above-mentioned multiple frames of images respectively, so as to adjust the illumination in each frame of the multiple frames of images. Optionally, in this embodiment, different adjustment factors can be determined by the average of the RGB values ​​of each frame of the multiple frames of images corresponding to the multiple illumination scenes included in the above-mentioned environmental information. Optionally, if the average of the RGB values ​​of each frame of the above-mentioned multiple frames of images is greater than a preset threshold, the value of the above-mentioned adjustment factor is less than 1, and if the average of the RGB values ​​of each frame of the above-mentioned multiple frames of images is less than or equal to the threshold, the value of the above-mentioned adjustment factor is greater than or equal to 1. It can be understood that if the mean value of the RGB value of each frame image in the above-mentioned multi-frame image is greater than the preset threshold, it means that the above-mentioned multi-frame image is a multi-frame image in an over-exposed scene, and the illumination of each frame image in the multi-frame image needs to be weakened; if the mean value of the RGB value of each frame image in the above-mentioned video frame sequence is less than or equal to the preset threshold, it means that the above-mentioned multi-frame image is a multi-frame image in an under-exposed scene, and the illumination of each frame image in the multi-frame image needs to be enhanced. Furthermore, the specific value of the above-mentioned adjustment factor can be determined according to the difference between the mean value of the RGB value of each frame image in the above-mentioned multi-frame image and the preset threshold, as long as the product value of the mean value of the RGB of each frame image and the above-mentioned adjustment factor is close to the preset threshold. For example, if the preset threshold is 128, and the mean value of the RGB value of each frame image in the above-mentioned multi-frame image is 50, the value of the adjustment factor can be 2.0, 2.3, 2.5, 2.53, etc. For another example, if the preset threshold is 120, and the average value of the RGB value of each frame of the above multiple frames of images is 200, then the value of the adjustment factor can be 0.5, 0.56, 0.58, 0.6, and so on.

[0056] In this embodiment, the server uses different adjustment factors to accurately adjust the RGB value of each frame image in the multiple frames of images included in the environmental information during the vehicle driving process, ensuring the accuracy of each adjusted frame image, so that each adjusted frame image can be accurately fused to obtain a target frame image with higher accuracy, thereby improving the accuracy of the target frame image.

[0057] In the above scenario where the target frame image is input into a preset recognition model to obtain a scene recognition result corresponding to the environmental information during vehicle driving, the above recognition model includes a feature extraction layer and a recognition layer. In one embodiment, Figure 5 As shown, the above S302 includes:

[0058] S501, input the target frame image into the feature extraction layer to obtain the features of the target frame image.

[0059] Specifically, the above-mentioned preset recognition model includes a feature extraction layer and a recognition layer. The server can first input the target frame image obtained above into the feature extraction layer of the recognition model, and extract features of the target frame image through the feature extraction layer to obtain features of the target frame image. Optionally, the feature extraction layer of the recognition model can include multiple convolutional layers, and the server can extract features of the target frame image through the multiple convolutional layers to obtain features of the target frame image. Optionally, the features of the target frame image obtained by the server can include one feature or multiple features.

[0060] S502, using different adjustment factors to adjust the illumination in the features of the target frame image to obtain adjusted features.

[0061] It should be noted that, in the present embodiment, if the features of the target frame image obtained by the server include multiple features, the server may use different adjustment factors to adjust the illumination in the features of each target frame image respectively, and then fuse the adjusted features of each target frame image to obtain the above-mentioned adjusted features. Similarly, in the present embodiment, the server may use different adjustment factors to adjust the RGB values ​​in the features of the target frame image to adjust the illumination in the features of the target frame image. It should be noted that the adjustment factor used in the present embodiment to adjust the illumination in the features of the target frame image is the same as the value of the adjustment factor used in the above-mentioned embodiment. For the value of the adjustment factor, please refer to the description in the above-mentioned embodiment, which will not be repeated in this embodiment.

[0062] S503, fusing the adjusted features to obtain fused features.

[0063] Optionally, the server may fuse the adjusted features according to the generation order of the adjusted features to obtain the fused features. Optionally, the fusion method for fusing the adjusted features may be a stacking fusion method, an averaging fusion method, or other fusion methods.

[0064] S504, inputting the fused features into the recognition layer to obtain a scene recognition result corresponding to the environmental information.

[0065] Specifically, in this embodiment, the server inputs the obtained fusion features into the recognition layer of the recognition model, and obtains the scene recognition result corresponding to the video frame sequence through the recognition layer of the recognition model. Exemplarily, taking the multi-frame images collected as multi-frame images of pedestrians running on the road under multiple lighting scenes as an example, the scene recognition result corresponding to the environmental information obtained by the server through the recognition layer of the recognition model is that the pedestrians on the road are running.

[0066] In this embodiment, the server inputs the above-mentioned target frame image into the feature extraction layer of the recognition model, and the features of the target frame image can be obtained through the feature extraction layer, so that different adjustment factors can be used to adjust the illumination in the features of the target frame image, and the features of the target frame image can be adjusted in different dynamic ranges, and the scene features under different illuminations can be mined from multiple degrees, and then the adjusted features are fused, so that the obtained fused features can include scene features under multiple different illuminations, and the obtained fused features can be input into the recognition layer of the preset recognition model, so that the scene corresponding to the environmental information obtained during the vehicle driving process can be accurately identified, thereby improving the accuracy of the scene recognition results corresponding to the environmental information obtained during the vehicle driving process.

[0067] In some scenarios, the server may also detect the target frame image through a detection model, or may also segment the target in the environmental information through a segmentation model. Based on the above embodiments, in one embodiment, Figure 6 As shown, the above method also includes:

[0068] S601, inputting the target frame image into a preset target detection model to perform target detection, and obtaining a target detection result corresponding to the target frame image.

[0069] Optionally, the target detection result corresponding to the target frame image obtained by the server may be a pedestrian on the road, or may be a vehicle traveling on the road, or may be a bicycle on the road, etc. Optionally, the target detection result corresponding to the target frame image obtained by the server may be marked in the target frame image in the form of a target detection box. Optionally, the preset target detection model may be any one of the neural network models of network structures such as VGG, ResNet, and I3D.

[0070] S602, input the target detection result and the target frame image into a preset segmentation model to obtain the segmentation result of the target in the environmental information.

[0071] Specifically, after the server obtains the target detection result corresponding to the target frame image, the target detection result and the target frame image can be input into a preset segmentation model to segment the target detected in the environmental information to obtain the segmentation result of the target in the environmental information. For example, the server can segment any target in the sky, streets, pedestrians on the road, and vehicles on the road in the environmental information. Optionally, the preset segmentation model can be any neural network model in the network structure such as VGG, ResNet, I3D, etc.

[0072] In this embodiment, the server inputs the target frame image into a preset target detection model, and can perform accurate target detection on the target frame image, and obtain a target detection result corresponding to the target frame image with higher accuracy. The obtained target detection result and the target frame image can then be input into a preset segmentation model, and the environmental information acquired during the vehicle driving process can be accurately segmented, thereby accurately obtaining the segmentation result of the target in the environmental information.

[0073] An embodiment of the present disclosure is introduced below in conjunction with a specific video recognition scenario. The method includes the following steps:

[0074] S1, obtaining environmental information during vehicle driving; the environmental information includes multiple frames of images corresponding to multiple lighting scenes.

[0075] S2, using different adjustment factors to adjust the RGB value of each frame image in the multiple frame images, and obtain the adjusted frame image corresponding to each adjustment factor; the adjustment factor is determined according to the average value of the RGB value of each frame image in the environmental information; if the average value of the RGB value of each frame image in the multiple frame images is greater than a preset threshold, the value of the adjustment factor is less than 1; if the average value of the RGB value of each frame image in the multiple frame images is less than or equal to the preset threshold, the value of the adjustment factor is greater than or equal to 1.

[0076] S3, fusing the adjusted frame images to obtain a target frame image.

[0077] S4, inputting the target frame image into the feature extraction layer in the preset recognition model to obtain the features of the target frame image.

[0078] S5, using different adjustment factors to adjust the illumination in the features of the target frame image to obtain adjusted features.

[0079] S6, fusing the adjusted features to obtain fused features.

[0080] S7, inputting the fused features into the recognition layer in the preset recognition model to obtain the scene recognition result corresponding to the environmental information.

[0081] S8, executing corresponding driving operations according to the scene recognition result.

[0082] S9, inputting the target frame image into a preset target detection model to perform target detection, and obtaining a target detection result corresponding to the target frame image.

[0083] S10, inputting the target detection result and the target frame image into a preset segmentation model to obtain the segmentation result of the target in the environmental information.

[0084] For the working principle of the scene recognition method provided in this embodiment, please refer to the detailed description in the above embodiment, which will not be described in detail in this embodiment.

[0085] It should be understood that although Figure 2-Figure 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 6 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0086] In one embodiment, Figure 7 As shown, a scene recognition device is provided, comprising: a first acquisition module, a second acquisition module and an execution module, wherein:

[0087] The first acquisition module is used to acquire environmental information during the driving process of the vehicle; the environmental information includes multiple frames of images corresponding to multiple lighting scenes;

[0088] The second acquisition module is used to obtain the scene recognition result corresponding to the environmental information; the scene recognition result is obtained by processing each frame of the multiple frames of images using different adjustment factors and a preset recognition model;

[0089] The execution module is used to perform corresponding driving operations according to the scene recognition results.

[0090] The scene recognition device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0091] Based on the above embodiment, optionally, the above second acquisition module includes: an adjustment unit and an acquisition unit, wherein:

[0092] The adjustment unit is used to adjust the illumination in each frame of the multiple frames of images by using different adjustment factors to obtain a target frame of images.

[0093] The acquisition unit is used to input the target frame image into a preset recognition model to obtain a scene recognition result corresponding to the environmental information.

[0094] The scene recognition device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0095] On the basis of the above embodiment, optionally, the above adjustment unit is used to use different adjustment factors to adjust the illumination in each frame image of the multiple frames to obtain an adjusted frame image corresponding to each adjustment factor; and fuse the adjusted frame images to obtain a target frame image.

[0096] The scene recognition device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0097] On the basis of the above embodiment, optionally, the above adjustment unit is used to adjust the RGB value of each frame image in the multiple frame images by using different adjustment factors to obtain each adjusted frame image.

[0098] Optionally, the adjustment factor is determined according to the average of the RGB values ​​of each frame image in the environmental information.

[0099] Optionally, if the average of the RGB values ​​of each frame of the multiple frames of images is greater than a preset threshold, the value of the adjustment factor is less than 1;

[0100] If the average of the RGB values ​​of each frame of the multiple frames of images is less than or equal to the preset threshold, the value of the adjustment factor is greater than or equal to 1.

[0101] The scene recognition device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0102] Based on the above embodiment, optionally, the recognition model includes a feature extraction layer and a recognition layer; the acquisition unit is used to input the target frame image into the feature extraction layer to obtain the features of the target frame image; use different adjustment factors to adjust the illumination in the features of the target frame image to obtain adjusted features; fuse the adjusted features to obtain fused features;

[0103] The fused features are input into the recognition layer to obtain the scene recognition results corresponding to the environmental information.

[0104] The scene recognition device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0105] Based on the above embodiment, optionally, the above device further includes: a detection module and a segmentation module, wherein:

[0106] The detection module is used to input the target frame image into a preset target detection model to perform target detection and obtain the target detection result corresponding to the target frame image.

[0107] The segmentation module is used to input the target detection results and the target frame image into a preset segmentation model to obtain the segmentation results of the target in the environmental information.

[0108] The scene recognition device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0109] For the specific definition of the scene recognition device, please refer to the definition of the scene recognition method above, which will not be repeated here. Each module in the above-mentioned scene recognition device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the server in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0110] Figure 8 1 is a block diagram of a server 1400 according to an exemplary embodiment. Figure 8The server 1400 includes a processing component 1420, which further includes one or more processors, and a memory resource represented by a memory 1422 for storing instructions or computer programs, such as applications, that can be executed by the processing component 1420. The application stored in the memory 1422 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1420 is configured to execute instructions to perform the above-mentioned scene recognition method.

[0111] The server 1400 may also include a power supply component 1424 configured to perform power management of the server 1400, a wired or wireless network interface 1426 configured to connect the server 1400 to a network, and an input / output (I / O) interface 1428. The server 1400 may operate based on an operating system stored in the memory 1422, such as Windows 1414ServerTM, Mac O14XTM, UnixTM, LinuxTM, FreeB14DTM or the like.

[0112] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 1422 including instructions, which can be executed by a processor of the server 1400 to perform the above method. The storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0113] In an exemplary embodiment, a computer program product is also provided, and when the computer program is executed by a processor, the above method can be implemented. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, part or all of the above method can be implemented in whole or in part according to the process or function described in the embodiment of the present disclosure.

[0114] Exemplarily, the present application embodiment discloses TS1, a scene recognition method, characterized in that the method comprises:

[0115] Acquire environmental information during vehicle driving; the environmental information includes multiple frames of images corresponding to multiple lighting scenes;

[0116] Obtaining a scene recognition result corresponding to the environmental information; the scene recognition result is obtained by processing each frame of the multiple images using different adjustment factors and a preset recognition model;

[0117] The corresponding driving operation is performed according to the scene recognition result.

[0118] TS2. The method as described in TS1, wherein the scene recognition result is obtained based on the following steps:

[0119] Using different adjustment factors to adjust the illumination in each frame of the multiple frames to obtain a target frame image;

[0120] The target frame image is input into a preset recognition model to obtain a scene recognition result corresponding to the environmental information.

[0121] TS3. The method as described in TS2, wherein the step of using different adjustment factors to adjust the illumination in each frame of the multiple frames to obtain a target frame image comprises:

[0122] Using different adjustment factors to adjust the illumination in each frame of the multiple frames of images to obtain an adjusted frame image corresponding to each adjustment factor;

[0123] The adjusted frame images are fused to obtain the target frame image.

[0124] TS4. The method as described in TS3, wherein the step of using different adjustment factors to adjust the illumination in each frame of the multiple frames to obtain an adjusted frame image corresponding to each adjustment factor comprises:

[0125] Different adjustment factors are used to adjust the RGB value of each frame image in the multiple frames of images to obtain each adjusted frame image.

[0126] TS5. A method as described in TS4, wherein the adjustment factor is determined based on the average of the RGB values ​​of each frame image in the environmental information.

[0127] TS6. The method of TS5, wherein the value of the adjustment factor is determined based on the following steps:

[0128] If the average of the RGB values ​​of each frame of the multiple frames of images is greater than a preset threshold, the value of the adjustment factor is less than 1;

[0129] If the average of the RGB values ​​of each frame of the multiple frames of images is less than or equal to the preset threshold, the value of the adjustment factor is greater than or equal to 1.

[0130] TS7. The method of TS2, wherein the recognition model comprises a feature extraction layer and a recognition layer; and the step of inputting the target frame image into a preset recognition model to obtain a scene recognition result corresponding to the environmental information comprises:

[0131] Inputting the target frame image into the feature extraction layer to obtain the features of the target frame image;

[0132] Using the different adjustment factors, adjusting the illumination in the features of the target frame image to obtain adjusted features;

[0133] Fusing the adjusted features to obtain fused features;

[0134] The fused features are input into the recognition layer to obtain a scene recognition result corresponding to the environmental information.

[0135] TS8. The method of TS2, wherein the segmentation result of the target in the environmental information is obtained based on the following steps:

[0136] Inputting the target frame image into a preset target detection model to perform target detection, and obtaining a target detection result corresponding to the target frame image;

[0137] The target detection result and the target frame image are input into a preset segmentation model to obtain a segmentation result of the target in the environmental information.

[0138] TS9. A scene recognition device, characterized in that the device comprises:

[0139] A first acquisition module is used to acquire environmental information during the driving process of the vehicle; the environmental information includes multiple frames of images corresponding to multiple lighting scenes;

[0140] A second acquisition module is used to acquire a scene recognition result corresponding to the environmental information; the scene recognition result is obtained by processing each frame of the multiple images using different adjustment factors and a preset recognition model;

[0141] An execution module is used to execute corresponding driving operations according to the scene recognition result.

[0142] TS10. A server comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in TS1 to TS8 when executing the computer program.

[0143] TS11. A storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of any one of the methods described in TS1 to TS8 are implemented.

[0144] TS12. A computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the steps of any one of the methods described in TS1-TS8 are implemented.

[0145] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the embodiments of the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0146] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above-described embodiments only express several implementation methods of the embodiments of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the embodiments of the present disclosure, and these all belong to the protection scope of the embodiments of the present disclosure. Therefore, the protection scope of the patent of the embodiments of the present disclosure shall be subject to the attached claims.

Claims

1. A scene recognition method, characterized in that: The method comprises: Acquire environmental information during vehicle driving; the environmental information includes multiple frames of images corresponding to multiple lighting scenes; the multiple lighting scenes include at least one scene of changing from day to night, from sunny to cloudy, from sunny to rainy, from cloudy to rainy, and entering or exiting a tunnel; Obtaining a scene recognition result corresponding to the environmental information; the scene recognition result is obtained by processing each frame of the multiple images using different adjustment factors and a preset recognition model; the adjustment factor is determined according to the average of the RGB values ​​of each frame of the environmental information; The corresponding driving operation is performed according to the scene recognition result.

2. The method according to claim 1, characterized in that The scene recognition result is obtained based on the following steps: Using different adjustment factors to adjust the illumination in each frame of the multiple frames to obtain a target frame image; The target frame image is input into a preset recognition model to obtain a scene recognition result corresponding to the environmental information.

3. The method according to claim 2, characterized in that The step of adjusting the illumination in each frame of the multiple frames of images by using different adjustment factors to obtain a target frame of images includes: Using different adjustment factors to adjust the illumination in each frame of the multiple frames of images to obtain an adjusted frame image corresponding to each adjustment factor; The adjusted frame images are fused to obtain the target frame image.

4. The method according to claim 3, characterized in that The step of using different adjustment factors to adjust the illumination in each frame of the multiple frames to obtain an adjusted frame image corresponding to each adjustment factor includes: Different adjustment factors are used to adjust the RGB value of each frame image in the multiple frames of images to obtain each adjusted frame image.

5. The method according to claim 1, characterized in that The value of the adjustment factor is determined based on the following steps: If the average of the RGB values ​​of each frame of the multiple frames of images is greater than a preset threshold, the value of the adjustment factor is less than 1; If the average of the RGB values ​​of each frame of the multiple frames of images is less than or equal to the preset threshold, the value of the adjustment factor is greater than or equal to 1.

6. The method according to claim 2, characterized in that The recognition model includes a feature extraction layer and a recognition layer; the target frame image is input into a preset recognition model to obtain a scene recognition result corresponding to the environmental information, including: Inputting the target frame image into the feature extraction layer to obtain the features of the target frame image; Using the different adjustment factors, adjusting the illumination in the features of the target frame image to obtain adjusted features; Fusing the adjusted features to obtain fused features; The fused features are input into the recognition layer to obtain a scene recognition result corresponding to the environmental information.

7. The method according to claim 2, characterized in that The segmentation result of the target in the environmental information is obtained based on the following steps: Inputting the target frame image into a preset target detection model to perform target detection, and obtaining a target detection result corresponding to the target frame image; The target detection result and the target frame image are input into a preset segmentation model to obtain a segmentation result of the target in the environmental information.

8. A scene recognition device, characterized in that: The device comprises: A first acquisition module is used to acquire environmental information during the driving process of the vehicle; the environmental information includes multiple frames of images corresponding to multiple lighting scenes; the multiple lighting scenes include at least one scene of changing from day to night, from sunny to cloudy, from sunny to rainy, from cloudy to rainy, and entering or exiting a tunnel; A second acquisition module is used to obtain a scene recognition result corresponding to the environmental information; the scene recognition result is obtained by processing each frame of the multiple images using different adjustment factors and a preset recognition model; the adjustment factor is determined according to the average value of the RGB value of each frame of the environmental information; An execution module is used to execute corresponding driving operations according to the scene recognition result.

9. A server comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images