Method, apparatus, device, storage medium and program product for scene detection

Abstract

According to an embodiment of the present disclosure, a method, device, equipment, storage medium and program product for scene detection are provided. The method comprises: acquiring, by using a plurality of sensors deployed on an autonomous vehicle, a plurality of perception data associated with a user riding the autonomous vehicle, the plurality of perception data at least comprising video data and audio data; determining, by using a plurality of encoders corresponding to the plurality of perception data, a plurality of encoding features corresponding to the plurality of perception data; determining a fused perception feature by fusing the plurality of encoding features; and determining a scene classification associated with the user based on the fused perception feature. Thus, embodiments of the present disclosure can more comprehensively refer to the plurality of perception data, improve the accuracy of determining the scene classification associated with the user, and thus improve the reliability of the autonomous vehicle. Further, based on such a scene classification, embodiments of the present disclosure can subsequently quickly detect an abnormal event associated with the user, further improving the reliability of the autonomous vehicle.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for scene detection. Background Technology

[0002] Autonomous driving is a technology that uses computers to replace human drivers, perceiving the vehicle's surroundings, planning its trajectory, and controlling it to reach its destination. Passengers riding in autonomous vehicles may experience a completely unmanned environment, meaning the cabin contains only passengers, without a driver or safety personnel. In such scenarios, any physical discomfort or emotional distress experienced by passengers may not be detected or addressed promptly, potentially leading to serious safety issues. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for scene detection is provided. The method includes: acquiring multiple sensor data associated with a user riding in an autonomous vehicle using multiple sensors deployed on the vehicle, the multiple sensor data including at least video data and audio data; determining multiple encoded features corresponding to the multiple sensor data using multiple encoders; determining fused sensor features by fusing the multiple encoded features; and determining a scene classification associated with the user based on the fused sensor features.

[0004] In a second aspect of this disclosure, a scene detection apparatus is provided. The apparatus includes: an acquisition module configured to acquire multiple sensor data associated with a user riding in an autonomous vehicle using multiple sensors deployed on the vehicle, the multiple sensor data including at least video data and audio data; a first determination module configured to determine multiple encoded features corresponding to the multiple sensor data using multiple encoders; a second determination module configured to determine fused sensor features by fusing the multiple encoded features; and a third determination module configured to determine a scene classification associated with the user based on the fused sensor features.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A schematic diagram illustrating an example process for scene detection according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic structural block diagram of an example device for scene detection according to certain embodiments of the present disclosure is shown; and

[0013] Figure 4 A block diagram of an apparatus capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0018] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0019] As briefly mentioned earlier, during the autonomous driving process of autonomous vehicles, unexpected situations such as passenger discomfort or emotional agitation among passengers may not be detected in time. Consequently, these abnormal events cannot be resolved promptly, potentially leading to serious consequences and affecting passenger safety. Some technical solutions equip autonomous vehicles with cameras, allowing onboard staff to monitor the situation and detect these anomalies. However, due to limitations in manpower and other factors, this approach makes it difficult to detect these anomalies in a timely manner, and some events are easily missed, posing significant safety risks to passengers riding in autonomous vehicles.

[0020] Based on this, embodiments of this disclosure propose a scene detection scheme. According to this scheme, multiple sensors deployed on an autonomous vehicle can be used to acquire various perceptual data associated with the user riding in the autonomous vehicle. These multiple perceptual data include at least video and audio data. Further, multiple encoders corresponding to the multiple perceptual data can be used to determine multiple encoded features corresponding to the multiple perceptual data. Further, multiple encoded features can be fused to determine fused perceptual features. Further, based on the fused perceptual features, a scene classification associated with the user can be determined.

[0021] In this way, embodiments of the present disclosure can acquire multiple sensory data associated with users riding in autonomous vehicles from multiple sensors, reducing the probability of missing perceptions of user-related events in autonomous vehicles; furthermore, embodiments of the present disclosure can fuse multiple coded features corresponding to multiple sensory data to more accurately determine the scene classification associated with users, thereby enabling faster detection of abnormal scenes such as passenger discomfort, improving the reliability of autonomous vehicles, and making it safer for users riding in such autonomous vehicles.

[0022] Therefore, the embodiments of this disclosure can more comprehensively reference various sensing data, improving the accuracy of determining the scene classification associated with the user, thereby improving the reliability of autonomous vehicles. Furthermore, based on such scene classification, the embodiments of this disclosure can subsequently detect abnormal events associated with the user more quickly, reducing user safety risks and further improving the reliability of autonomous vehicles.

[0023] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0024] Example Environment

[0025] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. As shown, environment 100 may include vehicle 110. Environment 100 can be applied to autonomous driving scenarios, assisted driving scenarios, intelligent transportation scenarios, etc. Vehicle 110 may be an autonomous vehicle, etc. An autonomous vehicle is a means of transportation with autonomous driving capability (or driverless driving capability), also known as a driverless car, autonomous driving vehicle, etc. For ease of understanding, this disclosure uses vehicle 110 as an example for explanation and illustration, but is not limited thereto.

[0026] In some scenarios, users can access travel services provided by vehicle 110 through travel apps. In other scenarios, vehicle 110 may also be referred to as a driverless taxi or robotaxi. During the process of providing travel services to users, vehicle 110 may be equipped with a safety operator. Vehicle 110 may also operate in an unmanned state. Vehicle 110 may also be equipped with one or more display devices inside the vehicle to provide human-machine interaction functions.

[0027] In some scenarios, at least one user, such as user A, may be riding in vehicle 110. Vehicle 110 may be equipped with various sensors to acquire perception data. For example, electronic device 130 uses the sensors deployed in vehicle 110 to acquire perception data 120 associated with user A. Based on such perception data 120, electronic device 130 can determine the scenario classification 140 associated with user A.

[0028] Such an electronic device 130 may include a terminal and / or a server. Such a terminal may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 130 may also support any type of user-facing interface (such as "wearable" circuitry).

[0029] Such servers can be standalone physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Servers can include, for example, computing systems / servers such as mainframes, edge computing nodes, and computing devices in cloud environments, etc.

[0030] The electronic device 130 can be installed in the vehicle 110, or deployed in any electronic unit of the vehicle 110 (such as sensors, control units, etc.), or it can exist independently of the vehicle 110.

[0031] When the electronic device 130 and the vehicle 110 exist independently, a communication connection can be established between the vehicle 110 and the electronic device 130. This communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth connections, mobile network connections, Universal Serial Bus connections, and Wi-Fi connections; the embodiments of this disclosure are not limited in this respect. Based on this, the vehicle 110 and the electronic device 130 can achieve signaling interaction through their communication connection.

[0032] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0033] Example process

[0034] Figure 2 A flowchart of an example process 200 for scene detection according to some embodiments of the present disclosure is shown. Process 200 can be implemented at electronic device 130. References are made below. Figure 1 Describe the process 200.

[0035] like Figure 2 As shown in block 210, electronic device 130 uses multiple sensors deployed on the autonomous vehicle to acquire multiple perceptual data associated with the user riding in the autonomous vehicle. The multiple perceptual data includes at least video data and audio data.

[0036] Taking an autonomous vehicle 110 as an example, and user A as a passenger in vehicle 110, vehicle 110 can be equipped with various sensors such as cameras, microphones, and radar. As examples, such sensors include cameras, microphones, and radar. For instance, electronic device 130 can utilize the cameras and other sensors deployed in vehicle 110 to acquire image data, video data, etc. Electronic device 130 can utilize the microphones and other sensors deployed in vehicle 110 to acquire audio data, etc. Electronic device 130 can utilize radar sensors (e.g., lidar, millimeter-wave radar) deployed in vehicle 110 to acquire motion information of objects (e.g., point cloud data, etc.). As examples, such motion information can indicate changes in the attitude of objects. Such point cloud data can be a data structure representing a set of points in three-dimensional space. Each point contains its coordinates in three-dimensional space (usually X, Y, Z coordinates), and sometimes additional information such as speed, distance, and direction.

[0037] As examples, such sensors can be deployed inside the interior space of vehicle 110, such as the cabin. Thus, such sensors can perceive various types of sensory data associated with user A. For example, audio data containing user A's voice, video data containing user A's image / video information, point cloud data containing user A's motion information, and so on.

[0038] The following will use cameras, microphones, and radar as examples to further explain how electronic device 130 uses sensors deployed in vehicle 110 to acquire perception data associated with user A.

[0039] When the sensors deployed in vehicle 110 include cameras, such cameras can be deployed in the area where user A is seated to ensure that the raw image data collected contains image information of user A. The resolution of such cameras can meet a preset resolution. Such cameras can also be wide-angle cameras. Therefore, such cameras can operate stably under different lighting conditions (including direct sunlight and low-light environments). Such cameras can acquire images at an image acquisition frame rate greater than or equal to a preset frame rate, thereby improving the complete capture of subtle movements of user A and enhancing adaptability to rapidly changing scenes.

[0040] In some embodiments, the electronic device 130 may preprocess the raw image data acquired by the camera to improve the accuracy of video data obtained based on such raw image data. Specifically, the electronic device 130 may acquire raw image data acquired by the camera and adjust the raw image data to determine video data. Adjusting the raw image data may include: denoising the raw image data using a filter, and / or adjusting the illumination information of the raw image data.

[0041] As examples, such filters may include Gaussian filters (Gaussian filtering algorithm), etc. Electronic device 130 can automatically adjust the parameters of the Gaussian filter based on the noise level of the original image data, thereby removing horizontal stripe noise, color cast noise, etc., from the original image data to improve the quality of the original image data.

[0042] As examples, the illumination of different image regions in the original image data may vary, meaning the original image data suffers from uneven illumination. Therefore, the electronic device 130 can adjust the histogram distribution of the original image data. Based on this adjustment, the electronic device 130 can enhance the contrast of image regions with different illumination in the original image data, thereby reducing the illumination differences between different image regions and avoiding subsequent feature loss or scene classification errors caused by such illumination differences.

[0043] When the sensors deployed in vehicle 110 include microphones, such microphones may include multiple microphones. As an example, such multiple microphones may include omnidirectional microphones. These multiple microphones may also have a sensitivity that meets a preset sensitivity and / or a frequency response range that meets a preset frequency response range. In this way, such multiple microphones can effectively collect audio from various directions and distances within the interior space of vehicle 110.

[0044] These multiple microphones can also be positioned in a pre-defined manner. For example, some of these microphones can be deployed in the front passenger compartment of vehicle 110, while others can be deployed in the rear passenger compartment. This allows for more comprehensive audio data collection and avoids interference between the microphones.

[0045] Multiple microphones can collect audio data at a frequency greater than or equal to a preset audio sampling frequency, allowing the acquired audio data to include more complete speech information. This audio sampling frequency can also be correlated with the image acquisition frame rate, ensuring that the acquired audio data and image data are synchronized, thus improving the accuracy of subsequent feature fusion.

[0046] In some embodiments, the audio data acquired by sensors such as microphones can be raw audio data. Further, the electronic device 130 can preprocess such raw audio data to obtain audio data for subsequent feature extraction. Such audio preprocessing methods may include adaptive filtering algorithms and / or adaptive echo cancellation methods, etc. For example, such adaptive filtering algorithms can reduce noise in the raw audio data to suppress environmental noise of the vehicle 110 (e.g., vehicle driving noise, external physical environmental noise, etc.). Such adaptive echo cancellation algorithms can eliminate echo data in the raw audio data to avoid echo problems caused by the limited space inside the vehicle. Thus, the electronic device 130 can improve the clarity of the audio data obtained from the preprocessing of the raw audio data, thereby improving the accuracy of subsequent feature extraction from the audio data.

[0047] In this way, electronic device 130 can acquire more accurate multi-sensory data.

[0048] In box 220, electronic device 130 uses multiple encoders corresponding to multiple types of sensor data to determine multiple coded features corresponding to the multiple types of sensor data.

[0049] As examples, such encoders can include neural network-based encoders. Such encoders can include, for example, pose feature encoders, facial feature encoders, audio feature encoders (e.g., speech recognition models based on a combination of convolutional neural networks and recurrent neural networks), and so on.

[0050] In some embodiments, the electronic device 130 may determine, based on video data, posture features and / or facial expression features associated with user A; and determine, based on posture features and / or facial expression features, video coding features corresponding to the video data.

[0051] As examples, electronic device 130 can identify the positional information of multiple joints associated with user A (e.g., head, shoulder, elbow, wrist, knee, ankle, etc.) based on video data. Based on this positional information, electronic device 130 can obtain the posture characteristics of user A.

[0052] As examples, electronic device 130 can locate multiple facial feature points (e.g., eyes, eyebrows, mouth, etc.) on user A's face based on video data. Furthermore, electronic device 130 can analyze the position and shape changes of these multiple facial feature points to obtain user A's facial expression characteristics. For example, if user A is unwell, user A's facial expression characteristics might indicate paleness, pain, or other states. As another example, if user A and other users in the same vehicle 110 are emotionally agitated, user A's possible facial expression characteristics might indicate anger, tension, or other emotions.

[0053] In some embodiments, the electronic device 130 may also add classification information to the facial expression features of user A based on a deep learning model (e.g., a convolutional neural network) to further improve the accuracy of subsequent scene classification associated with user A.

[0054] Thus, electronic device 130 can determine more accurate multiple coded features corresponding to various types of sensing data.

[0055] In box 230, electronic device 130 determines fused sensing features by fusing multiple coded features.

[0056] As examples, fusion can be achieved in various ways, including but not limited to concatenation, weighted summation, attention-based fusion, and so on. By fusing multiple encoded features, the determined fused perceptual features allow electronic device 130 to detect user A's state more comprehensively.

[0057] In some embodiments, the electronic device 130 can determine fused sensing features by concatenating multiple coded features. As some examples, the multiple coded features can be in vector form, and the electronic device 130 can concatenate the multiple coded features in a preset order (e.g., vector concatenation) to obtain such fused sensing features.

[0058] In some embodiments, the electronic device 130 may also fuse multiple encoded features based on an attention mechanism to determine fused perceptual features, wherein the weight information associated with the attention mechanism is determined based on a set of training data. For example, the training samples in such a set of training data may include multiple training encoded features and training weights corresponding to the multiple training encoded features, such training weights can be obtained by annotation through a labeling model or by other labeling methods.

[0059] As examples, such an attention mechanism can automatically learn the attention weights of multiple encoded features across multiple scenes. Thus, the electronic device 130 can input these multiple encoded features into such an attention network to obtain the weights assigned to each encoded feature by the attention mechanism, i.e., determine the weight information associated with the attention mechanism. Furthermore, based on this weight information, the electronic device 130 can perform calculations such as weighted summation on the multiple encoded features to determine such fused perceptual features, which can then be used for subsequent scene classification.

[0060] In box 240, electronic device 130 determines the scene classification associated with the user based on fused perception features.

[0061] As examples, the scenarios associated with user A may include normal scenarios, abnormal scenarios, and so on. Abnormal scenarios include, for example, scenarios where user A is physically unwell or emotionally agitated. It should be understood that such scenario classifications are not limited to these examples, and this disclosure does not exhaustively list such scenario classifications.

[0062] Furthermore, based on such fused perception features, electronic device 130 can detect user A from multiple angles, thereby enabling a more accurate scene classification associated with user A.

[0063] In some embodiments, such scene classification can be achieved using a classification model to improve the efficiency of determining the scene classification. Specifically, the electronic device 130 can provide fused sensing features to the classification model; and based on the output information of the classification model, determine the scene classification corresponding to the fused sensing features.

[0064] As examples, the classification model may include a neural network-based classification model, which can be pre-trained to accurately classify such fused perceptual features. Exemplarily, the output information of such a classification model may include classification labels indicating different scene classifications (i.e., different classification labels correspond to different scene classifications). Based on such classification labels, the electronic device 130 can accurately determine the scene classification corresponding to the fused perceptual features.

[0065] In some embodiments, the electronic device 130 can acquire reference information associated with user A; and based on the reference information and output information, determine the scene classification corresponding to the fused perceptual features. As examples, such reference information includes, but is not limited to, route information (e.g., origin and destination) uploaded by user A, and the time information of the vehicle ride. Thus, the electronic device 130 can determine whether user A's current route is to a specific location such as a hospital. Alternatively, the electronic device 130 can determine whether user A's current ride is at night or in the evening, etc. Such reference information can be used to increase or decrease the probability that user A is associated with scene classifications such as physical discomfort.

[0066] In some embodiments, the acquisition and use of the above reference information are carried out with the user's knowledge and permission, and all comply with relevant laws, regulations and related provisions.

[0067] In some embodiments, the electronic device 130 may adjust the confidence level determined by the classification model and associated with at least one preset scene classification based on reference information; and determine the scene classification corresponding to the fused perception features based on the adjusted confidence level.

[0068] As examples, if the reference information indicates whether user A's current route is to a special location such as a hospital, or whether user A's current travel time is evening or nighttime, the probability of user A being associated with such an unusual scenario category increases. Based on this, electronic device 130 can adjust the confidence level determined by the classification model and associated with at least one preset scenario category (e.g., increase such confidence level) based on the reference information, thereby determining a more accurate scenario category.

[0069] In some embodiments, the electronic device 130 can classify multiple perceptual features separately to obtain multiple initial scene classification results corresponding to the multiple perceptual features; and fuse these multiple initial scene classification results to obtain a scene classification associated with user A. As examples, such classification can be implemented using models, mapping tables, etc. The fusion of such multiple initial scene classification results can refer to the specific implementation of the fusion of multiple perceptual features described above, and will not be repeated here. Thus, the electronic device 130 can improve scene detection efficiency.

[0070] In some embodiments, after determining a scenario category associated with user A, if such a scenario category indicates that user A is in an abnormal state (e.g., physical discomfort, emotional agitation with other users A, etc.), electronic device 130 can request assistance from a remote device. Specifically, electronic device 130 can send an assistance request associated with the scenario category to a remote device in response to the scenario category meeting preset conditions. As examples, such preset conditions may indicate that user A is in an abnormal state (e.g., physical discomfort, emotional agitation with other users A, etc.). Thus, the abnormal state of user A can be addressed promptly, thereby reducing the security risks posed by user A.

[0071] Based on this approach, embodiments of this disclosure can acquire multiple sensory data associated with users riding in autonomous vehicles from multiple sensors, reducing the probability of missing perceptions of user-related events in autonomous vehicles. Furthermore, embodiments of this disclosure can fuse multiple coded features corresponding to various sensory data to more accurately determine the scene classification associated with the user, thereby enabling faster detection of abnormal scenarios such as passenger discomfort, improving the reliability of autonomous vehicles, and making it safer for users riding in such autonomous vehicles.

[0072] Therefore, the embodiments of this disclosure can more comprehensively reference various sensing data, improving the accuracy of determining the scene classification associated with the user, thereby improving the reliability of autonomous vehicles. Furthermore, based on such scene classification, the embodiments of this disclosure can subsequently detect abnormal events associated with the user more quickly, reducing user safety risks and further improving the reliability of autonomous vehicles.

[0073] Example devices and equipment

[0074] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 3 A schematic structural block diagram of a scene detection device 300 according to certain embodiments of the present disclosure is shown. The device 300 may be implemented as or included in an electronic device 130. Various modules / components in the device 300 may be implemented by hardware, software, firmware, or any combination thereof.

[0075] like Figure 3As shown, the device 300 includes an acquisition module 310 configured to acquire multiple sensor data associated with a user riding in the autonomous vehicle using multiple sensors deployed on the autonomous vehicle, the multiple sensor data including at least video data and audio data; a first determination module 320 configured to determine multiple coding features corresponding to the multiple sensor data using multiple encoders; a second determination module 330 configured to determine fused sensor features by fusing the multiple coding features; and a third determination module 340 configured to determine a scene classification associated with the user based on the fused sensor features.

[0076] In some embodiments, the second determining module 330 is further configured to: cascade multiple coded features to determine fused sensing features.

[0077] In some embodiments, the second determining module 330 is further configured to: fuse multiple encoded features based on an attention mechanism to determine fused perceptual features, wherein the weight information associated with the attention mechanism is determined based on a set of training data.

[0078] In some embodiments, the first determining module 320 is configured in one step to: determine, based on video data, gesture features and / or facial expression features associated with the user; and determine, based on gesture features and / or facial expression features, video coding features corresponding to the video data.

[0079] In some embodiments, the acquisition module 310 is configured in one step to: acquire raw image data captured by the camera; and adjust the raw image data to determine video data, wherein adjusting the raw image data includes: denoising the raw image data using a filter, and / or adjusting the illumination information of the raw image data.

[0080] In some embodiments, the multiple sensing data also include motion information acquired using radar sensors.

[0081] In some embodiments, the third determining module 340 is further configured to: provide fused sensing features to the classification model; and determine the scene classification corresponding to the fused sensing features based on the output information of the classification model.

[0082] In some embodiments, the third determining module 340 is further configured to: acquire reference information associated with the user; and determine the scene classification corresponding to the fused perception features based on the reference information and the output information.

[0083] In some embodiments, the third determining module 340 is further configured to: adjust the confidence level determined by the classification model and associated with at least one preset scene classification based on reference information; and determine the scene classification corresponding to the fused perception features based on the adjusted confidence level.

[0084] In some embodiments, the apparatus 300 further includes a request for assistance module configured to send an assistance request associated with the scene classification to a remote device in response to the scene classification meeting preset conditions.

[0085] The units included in device 300 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 300 may be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0086] Figure 4 A block diagram of an electronic device 400 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 4 The electronic device 400 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 4 The electronic device 400 shown can be used to achieve Figure 1 Electronic devices 110 or Figure 3 Device 300.

[0087] like Figure 4 As shown, electronic device 400 is in the form of a general-purpose electronic device. Components of electronic device 400 may include, but are not limited to, one or more processors or processing units 410, memory 420, storage device 430, one or more communication units 440, one or more input devices 450, and one or more output devices 460. Processing unit 410 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 420. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 400.

[0088] Electronic device 400 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 400, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 420 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 430 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 400.

[0089] Electronic device 400 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 4 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 420 may include computer program product 425 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0090] Communication unit 440 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 400 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 400 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0091] Input device 450 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 460 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 400 can also communicate with one or more external devices (not shown) via communication unit 440 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 400, or with any device that enables electronic device 400 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0092] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0093] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0094] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0095] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0097] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for scene detection, comprising: By utilizing multiple sensors deployed on the autonomous vehicle, various sensory data associated with the user riding in the autonomous vehicle are acquired, including at least video data and audio data. Using multiple encoders corresponding to the various types of sensory data, multiple coding features corresponding to the various types of sensory data are determined; By fusing the multiple encoded features, the fused perceptual features are determined; as well as Based on the fused perception features, the scene classification associated with the user is determined.

2. The method according to claim 1, wherein determining the fused perceptual features by fusing the plurality of encoded features includes: The multiple encoded features are cascaded to determine the fused sensing features.

3. The method according to claim 1, wherein determining the fused perceptual features by fusing the plurality of encoded features includes: Based on an attention mechanism, the multiple encoded features are fused to determine the fused perceptual features, wherein the weight information associated with the attention mechanism is determined based on a set of training data.

4. The method according to claim 1, wherein determining the plurality of encoded features corresponding to the plurality of perceived data includes: Based on the video data, determine the posture features and / or facial expression features associated with the user; as well as Based on the posture features and / or the facial expression features, determine the video encoding features corresponding to the video data.

5. The method according to claim 1, wherein acquiring the video data comprises: Acquire raw image data captured by the camera; as well as Adjusting the original image data to determine the video data, wherein adjusting the original image data includes: denoising the original image data using a filter, and / or adjusting the illumination information of the original image data.

6. The method according to claim 1, wherein the plurality of sensing data further includes motion information acquired using a radar sensor.

7. The method of claim 1, wherein determining the scene classification associated with the user based on the fused perception features comprises: Provide the fused perceptual features to the classification model; as well as Based on the output information of the classification model, the scene classification corresponding to the fused perception features is determined.

8. The method according to claim 7, wherein determining the scene classification corresponding to the fused perceptual features based on the output information of the classification model comprises: Obtain reference information associated with the user; as well as Based on the reference information and the output information, the scene classification corresponding to the fused perception features is determined.

9. The method according to claim 8, wherein determining the scene classification corresponding to the fused perception features based on the reference information and the output information comprises: Based on the reference information, adjust the confidence level determined by the classification model that is associated with at least one preset scene classification; as well as Based on the adjusted confidence level, the scene classification corresponding to the fused perception features is determined.

10. The method according to claim 1, further comprising: In response to the scene classification meeting preset conditions, an assistance request associated with the scene classification is sent to a remote device.

11. An apparatus for scene detection, comprising: The acquisition module is configured to acquire various sensor data associated with a user riding in the autonomous vehicle using multiple sensors deployed on the autonomous vehicle, the multiple sensor data including at least video data and audio data; The first determining module is configured to determine multiple encoded features corresponding to the multiple types of sensory data using multiple encoders corresponding to the multiple types of sensory data; The second determining module is configured to determine the fused sensing features by fusing the multiple coded features; as well as The third determining module is configured to determine the scene classification associated with the user based on the fused perception features.

12. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 10.

14. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 10.

Images