Ambient perception system for experiencing the environment through music

Abstract

The present invention discloses an environment perception system, which includes a memory and a first module and a second module. The memory is configured to store environment data, one or more music work templates and one or more mappings, wherein the environment data indicates at least one of a state, condition or change in the environment in which the environment perception system is located. The first module is configured to receive the environment data and store the environment data in the memory. The second module is configured to: based on the one or more music work templates and the one or more mappings, convert the environment data into a music signal, including modifying variables in the one or more music work templates based on the environment data; and based on the music signal, play the music work via an audio system to audibly indicate at least one of a state, condition or change in the environment.

Description

Technical Field

[0001] The information provided in this section is for the purpose of generally introducing the contents of the present disclosure. The work of the presently named inventors within the scope described in this section, and aspects of the description that may not otherwise constitute prior art at the time of filing, are neither explicitly nor implicitly admitted to be prior art against the present disclosure.

[0002] The present disclosure relates to situational awareness enhancement and warning systems. Background Art

[0003] The vehicle includes one or more torque-generating devices, such as an internal combustion engine and / or an electric motor, and is typically designed to be driven for a long period of time. The vehicle may be non-autonomous, or may be partially or fully autonomous. The occupants of the vehicle ride in the cabin (or passenger compartment) of the vehicle. The occupants in the vehicle may include a driver (or pilot) and one or more passengers. If the vehicle is fully autonomous, all occupants may be referred to as passengers.

[0004] The vehicle may include sensors for sensing the vehicle's surroundings. As an example, a camera may be used to capture images of the vehicle's external environment. The vehicle may include front-facing, rear-facing, and side-facing cameras. Other exemplary sensors for sensing the vehicle's surroundings include radar, sonar, and light detection and ranging (LIDAR) sensors. Summary of the invention

[0005] An environment perception system is provided, and includes a memory, a first module, and a second module. The memory is configured to store environment data, one or more music composition templates, and one or more mappings, wherein the environment data indicates at least one of a state, condition, or change in the environment in which the environment perception system is located. The first module is configured to receive the environment data and store the environment data in the memory. The second module is configured to: based on the one or more music composition templates and the one or more mappings, convert the environment data into a music signal, including modifying variables in the one or more music composition templates based on the environment data; and based on the music signal, play a music composition via an audio system to audibly indicate at least one of the state, condition, or change in the environment.

[0006] In other features, the system further comprises an audio system. The audio system is configured to receive the music signal and play the musical composition. The musical composition comprises instrumental music.

[0007] In other features, the musical composition also includes a vocal portion.

[0008] In other features, the memory is configured to store user preferences.The second module is configured to generate or modify at least one of the one or more musical composition templates or the one or more mappings based on the user preferences.

[0009] In other features, the system further comprises an interface configured to receive user input. The second module is configured to select the one or more musical composition templates based on the user input, based on a user preference indicated by the user input or an instruction included in the user input, to change a track of the musical composition.

[0010] In other features, the second module is configured to generate the music signal to include different musical phrases for different points of interest.

[0011] In other features, the second module is configured to generate the music signal to include different musical phrases for different environmental conditions.

[0012] In other features, the second module is configured to generate the music signal to include a subject to musically indicate a distance to one or more points of interest.

[0013] In other features, the second module is configured to generate the music signal to include different musical pieces for different instruments to musically indicate different points of interest.

[0014] In other features, the second module is configured to generate the music signal to include sound effects to musically indicate changes in different environmental states or conditions.

[0015] In other features, the second module is configured to generate the musical signal to include dissonant chords to musically indicate a hazardous condition.

[0016] In other features, the second module is configured to generate the music signal to include an extended tone or a discordant tone to musically indicate that a current state or condition does not match a user request.

[0017] In other features, the second module is configured to receive user input and adjust parameters of the music signal based on the user input. The parameters include at least one of a music style parameter, an instrument parameter, a pitch mapping parameter, a time control parameter, and a harmony parameter.

[0018] In other features, the second module is configured to adjust at least one of the tempo, pitch, volume, melody, or harmony of the music signal based on changes in the environmental data.

[0019] In other features, the environmental data is received from one of one or more sensors.

[0020] In other features, the environmental data is received from a wireless network or a distributed network.

[0021] In other features, the system further includes a transceiver configured to transmit the environmental data from the vehicle to a mobile network device. The memory is implemented in the vehicle. The second module is implemented in the mobile network device.

[0022] In other features, the system further includes a transceiver configured to transmit the music signal from the vehicle to a mobile network device. The memory and the second module are implemented in the vehicle. The audio system is implemented in or connected to the mobile network device.

[0023] In other features, a network device is provided and includes the above system and a transceiver. The transceiver is configured to receive the environmental data from at least one of a sensor, a vehicle, or a distributed network.

[0024] In other features, a vehicle is provided and includes the above system and the audio system. The audio system is configured to play music in the vehicle based on the music signal.

[0025] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims and drawings.The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.

[0026] The present disclosure also includes the following technical solutions.

[0027] Technical Solution 1. An environment perception system, comprising:

[0028] a memory configured to store environmental data, one or more musical composition templates, and one or more mappings, wherein the environmental data indicates at least one of a state, a condition, or a change in an environment in which the environmental awareness system is located;

[0029] A first module configured to receive the environmental data and store the environmental data in the memory; and

[0030] The second module is configured to

[0031] converting the ambient data into a music signal based on the one or more musical composition templates and the one or more mappings, including modifying variables in the one or more musical composition templates based on the ambient data, and

[0032] Based on the music signal, a musical composition is played via an audio system to audibly indicate at least one of a state, condition, or change in the environment.

[0033] Technical Solution 2. The system according to Technical Solution 1 further comprises the audio system, wherein:

[0034] The audio system is configured to receive the music signal and play the music piece; and

[0035] The musical work includes instrumental music.

[0036] Technical Solution 3. A system according to Technical Solution 2, wherein the musical work also includes a vocal part.

[0037] Technical Solution 4. The system according to Technical Solution 1, wherein:

[0038] The memory is configured to store user preferences; and

[0039] The second module is configured to generate or modify at least one of the one or more musical composition templates or the one or more mappings based on user preferences.

[0040] Technical Solution 5. The system according to Technical Solution 1 further comprises an interface configured to receive user input,

[0041] The second module is configured to select the one or more music work templates based on the user input, based on the user preference indicated by the user input or the instruction included in the user input, so as to change the track of the music work.

[0042] Technical Solution 6. The system according to Technical Solution 1, wherein the second module is configured to generate the music signal to include different musical phrases for different points of interest.

[0043] Technical Solution 7. The system according to Technical Solution 1, wherein the second module is configured to generate the music signal to include different music phrases for different environmental conditions.

[0044] Technical Solution 8. The system according to Technical Solution 1, wherein the second module is configured to generate the music signal to include a subject so as to musically indicate the distance from one or more points of interest.

[0045] Technical Solution 9. A system according to Technical Solution 1, wherein the second module is configured to generate the music signal to include different musical pieces for different instruments so as to indicate different points of interest musically.

[0046] Technical Solution 10. A system according to Technical Solution 1, wherein the second module is configured to generate the music signal to include sound effects so as to musically indicate changes in different environmental states or conditions.

[0047] Technical Solution 11. The system according to Technical Solution 1, wherein the second module is configured to generate the music signal to include dissonant chords so as to musically indicate a dangerous condition.

[0048] Technical Solution 12. A system according to Technical Solution 1, wherein the second module is configured to generate the music signal to include extended sounds or discordant sounds so as to musically indicate that the current state or condition does not match the user request.

[0049] Technical Solution 13. The system according to Technical Solution 1, wherein:

[0050] The second module is configured to receive user input and adjust a parameter of the music signal based on the user input; and

[0051] The parameters include at least one of a music style parameter, an instrument parameter, a pitch mapping parameter, a time control parameter, and a harmony parameter.

[0052] Technical Solution 14. A system according to Technical Solution 1, wherein the second module is configured to adjust at least one of the rhythm, pitch, volume, melody, or harmony of the music signal based on changes in the environmental data.

[0053] Technical Solution 15. The system according to Technical Solution 1, wherein the environmental data is received from one of one or more sensors.

[0054] Technical Solution 16. The system according to Technical Solution 1, wherein the environmental data is received from a wireless network or a distributed network.

[0055] Technical Solution 17. The system according to Technical Solution 1 further comprises a transceiver configured to transmit the environmental data from the vehicle to a mobile network device, wherein:

[0056] The memory is implemented in the vehicle; and

[0057] The second module is implemented in the mobile network device.

[0058] Technical Solution 18. The system according to Technical Solution 1 further comprises a transceiver configured to transmit the music signal from the vehicle to a mobile network device, wherein:

[0059] The memory and the second module are implemented in the vehicle; and

[0060] The audio system is implemented in the mobile network device or is connected to the mobile network device.

[0061] Technical Solution 19. A network device comprising:

[0062] The system according to technical solution 1; and

[0063] A transceiver is configured to receive the environmental data from at least one of a sensor, a vehicle, or a distributed network.

[0064] Technical Solution 20. A vehicle comprising:

[0065] The system according to technical solution 1; and

[0066] An audio system configured to play music within the vehicle based on the music signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The present disclosure will become more fully understood based on the detailed description and accompanying drawings, in which:

[0068] Figure 1 is an exemplary diagram of two tones played at specific time intervals;

[0069] Figure 2 is a functional block diagram of an example of a network device including a context awareness module according to the present disclosure;

[0070] Figure 3 is a functional block diagram of an example of an ambient notification system according to the present disclosure;

[0071] Figure 4 is a functional block diagram of an example of a vehicle including an environment perception module according to the present disclosure;

[0072] Figure 5 is a functional block diagram of an example of an ambient notification system that provides ambient data directly to a mobile device according to the present disclosure;

[0073] Figure 6 is a functional block diagram of an example of an environmental perception module and a memory according to the present disclosure;

[0074] Figure 7 Shows the operation Figure 6 The method of the environment perception module; and

[0075] Figure 8 is an exemplary diagram illustrating features associated with a musical representation of ambient data in accordance with the present disclosure.

[0076] In the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0077] In various applications, environmental data can be tracked, monitored, and acted upon. As an example, environmental data can be monitored to indicate to the driver of a vehicle that the vehicle is approaching an object (e.g., another vehicle). Environmental data can be shown to the driver via a display and / or audible sounds, such as showing a view behind the corresponding vehicle and generating a beep when backing up and approaching an object. As another example, a display on a mobile device or a display in a vehicle can show a map and provide instructions to reach a specific location and / or indicate when the user has arrived at a selected destination. As yet another example, status information of a vehicle system can be visually displayed to indicate when a specific system has a problem and / or when maintenance is required. This information can be displayed via a dashboard indicator and / or a head-up display.

[0078] In the commercial, government and academic fields, a large amount of environmental data is generated. It may be difficult to transmit such a large amount of data and / or corresponding conditions and / or parameters determined based on the data to a user in a manner that the user can recognize. The data may be associated with a large number of variables, and transmitting such a large number of variables to the user may be overwhelming and / or simply not feasible. If too many variables are reported to the user, the user may ignore some or all of the reported variables.

[0079] Some conventional methods for indicating environmental information include two-dimensional and three-dimensional graphics, animated videos, visual warning screens, icons, etc. The first step in organizing and simplifying the information to be reported to the user is to reduce the dimension using principal component analysis, self-organizing maps and other techniques. However, it is critical that the user understands the reported data without losing too much information.

[0080] Examples described herein include systems and environmental perception modules for converting environmental data and corresponding detected conditions and parameters into musical sounds. The musical sounds are used for dimensionality reduction in order to transmit various environmental information to the user in a simple and understandable format, such as mobile device or vehicle status information, external conditions, information about the user's surroundings, destination information, safety conditions, danger information, etc. These examples can be implemented via mobile network devices (or mobile devices), such as cellular phones, smart phones, tablet computers, laptops, wearable devices, vehicles (e.g., land-based vehicles, aircraft or ships), bicycles, etc. In one embodiment, the system is implemented to track, monitor and report geographic information system (GIS) point of interest (PoI) data. GIS PoI may include (i) the location of the network device implementing the reporting method disclosed herein, (ii) the destination selected by the user, (iii) a physical structure, such as a monument or building being passed, (iv) a park being passed, etc. Environmental data and / or PoIs may further refer to and / or indicate locations such as ATMs, bridges, cities, towns, entertainment venues, hazardous conditions (roads, weather, flooding, etc.), hotels, intersections and junctions, landmarks, medical facilities, parking structures, public safety (police stations, fire stations, weigh stations), railroad crossings, and restaurants. Other environmental data are described below.

[0081] Ambient data is converted to sound using musical compositions. This allows the consumption of ambient data to be expanded beyond text, graphics and other visual indicators and / or simple auditory tones to include musical tracks. These examples leverage the auditory abilities of humans (i.e., users of the system) to better inform users of their surroundings. This is helpful for people who are at least partially visually impaired, and enhances the experience for others who are not visually impaired. These examples allow users to listen to the user's surroundings in an informed manner through music, without necessarily verbal instructions.

[0082] During safe driving behavior, the driver's visual attention has been consumed by multiple inputs by observing the outside of the vehicle and regularly checking the rearview mirror and side mirrors, head-up display and other displays and visual indicators. These examples allow users to detect various aspects of the surrounding environment through music without visually viewing the rearview mirror or side mirror or looking down at the display or dashboard indicator. Representing environmental information as music is very suitable for unstructured data, which is usually imprecise and difficult to depict in a user-recognizable manner. Unstructured data can refer to data that is not easily related to other data and / or tabulated and does not have a specified relationship with other data and / or follows a standard pattern. Structured data can refer to data that has a specified relationship with other data and / or follows a structured pattern. In addition, environmental data from many different sources can be converted into music, and these data can be mentally visualized. As a result, users can mentally visualize situations based on the music heard by the user.

[0083] There are situations where it is useful to convert sensor data, operational data, and / or real-time input into musical sound. For example, Figure 1 The function Sin[x]+Sin[ x] is a diagram 100. This diagram may seem complicated to the human eye, but the sound with the corresponding waveform heard by the human ear is considered to be composed of two pure musical tones. Therefore, by converting the data into musical tones, the user can more easily grasp certain types of data.

[0084] These examples also include communicating various satisfying conditions and / or relationships between environmental data through music. These examples use sonification to enhance and / or replace visually communicated information to reduce the cognitive load of the driver, pilot, passenger and / or other user when operating and / or riding in a vehicle.

[0085] Figure 2A network device 200 is shown that includes a control module 202 having an environmental sensing module 204. The network device 200 can be a mobile network device or other network device, such as a desktop computer, a home assistant device (or a smart home controller hub), a vehicle or part thereof, etc. The environmental sensing module 204 obtains environmental data, converts the data into music, and plays the music via an audio system 206, which may include speakers, earbuds, etc. In one embodiment, the environmental data is played continuously until, for example, the audio playback of the music is disabled by the user. As further described below, the music can be played as background music or as music selected by the user and generated based on the user's preferences. The network device 200 may include a sensor 208, a memory 210, a display 212, and a transceiver 214. The sensor 208 may include a camera 216, an accelerometer 218, one or more temperature sensors (one temperature sensor 220 is shown), and / or other sensors 222. Other exemplary sensors are disclosed below.

[0086] The environment sensing module 204 can collect environment data from various sources, such as sensors 208, transceivers 214, other modules and / or memory 210. The environment data can be collected and stored in the memory 210. The network device 200 can communicate with other network devices via the transceiver 214 and receive the environment data. The environment sensing module 204 can also analyze the environment data to determine whether certain conditions exist, and store the conditions in the memory 210 as additional environment data. The memory 210 stores the environment data (represented as 230), the music piece 232 and the data to music mapping algorithm 234. The data to music mapping algorithm may include functions, tables, formulas, relationships, etc. to map the environment data to variable values. As used herein, music (or music) works refer to (i) a piece of music or a work (vocal and / or instrumental), and / or (ii) the structure of a piece of music. The music piece 232 includes a variable and encoded music piece (hereinafter referred to as a "music piece") having variables that can be modified based on the environment data. The variable values ​​are applied to the musical composition 232 to change the way the musical composition is played, as further described below.

[0087] Short Sentences

[0088] A musical composition may include one or more musical phrases. A musical phrase is a series of tones grouped together and having a sense of completeness and unity, like those found in a verb sentence and consisting of images, themes and units. Phrases can be combined to form melodies, phrases and larger parts of music. Phrases are substantial musical ideas that end with musical punctuation marks called cadences. Specific different musical phrases can be associated with PoIs and / or other environmental data. Musical phrases can be customer-created or selected from a database of pre-existing phrases (e.g., a ditty for an advertisement for a particular restaurant can be used to indicate that such a restaurant is nearby). Phrases can be embedded in background music, which can also be written or selected by the customer. Background music can be an existing piece of music, thereby providing listeners with pleasant and continuous ambient music, in which musical phrases fade in and out of the ambient music. As an example, in order to indicate to the user the presence of a PoI near an internal PoI of a larger PoI (e.g., a particular neighborhood of a city or different types of PoIs within a downtown area), a chain phrase can be used. A chain phrase is a single phrase consisting of a complex theme of musical ideas.

[0089] Purpose

[0090] The work may include a theme. As an example, a theme may be used to indicate the distance from the user's current location to a PoI. A theme (or subject) is a musical structure that includes short, continuous or often recurring musical images or shapes. Each theme has different characteristics and profiles. A different distinct theme may be assigned to each of the discrete, different PoIs. Adjacent PoIs may share a theme or have overlapping themes.

[0091] voice

[0092] As another example, different "voices" may be included in a musical composition and assigned to different PoIs. A voice may refer to the use of a particular instrument or set of instruments to indicate a particular PoI. Each particular PoI may be assigned a different set of one or more instruments. In one embodiment, each PoI is assigned a different part of an orchestra, in other words, a different set of instruments. The voice may be played so that a user can simultaneously hear portions of the music being played that correspond to two or more PoIs, such as a restaurant and a hotel, respectively.

[0093] As the number of sonified PoIs increases and is played, it may become increasingly difficult for a user to accurately detect any single sonified PoI. To minimize this problem, the amount of PoIs tracked may be limited, and the number of different instrument groups for each PoI tracked may also be limited. As an example, 3-5 PoIs may each include four instruments, such as a classical string quartet or a jazz quartet.

[0094] Automatic algorithms implemented by the context awareness module may be used to assign different PoI music phrases to different instruments. These algorithms may determine when to play various PoI music phrases based on configuration values. The PoI music phrases may be used to indicate the next "big city" approaching as it is within different distances (e.g., 100, 50, 25, 10 miles) of the moving vehicle. The volume or intensity of the phrase may increase as the PoI becomes closer to the user.

[0095] Other contents of the musical work

[0096] Sound effects such as panning the music speech to the left or right speaker and changing the volume of the music speech can be performed. Sound effects can be performed to maximize the personality of the audible elements in the music. Various effects can be provided and include arrangement, voice guidance, development of complete melodic phrases, etc. These various effects can have corresponding sonification parameters that are set to provide corresponding effects. Sound effects can be used to indicate where environmental information is located. For example, sound effects can be provided to indicate to the user "PoI passing by your left", "vehicle approaching quickly from the right", etc.

[0097] Dangerous road conditions, traffic jams, other road and traffic conditions, fog, slippery roads, winter storms, floods, fires, and / or other conditions detected may each be provided with a corresponding musical phrase. As another example, when there are no dangerous conditions or faults to be reported to the user, the overall tempo of the music being played may be slow and steady. When a dangerous condition occurs, the environment perception module 204 may increase the tempo of the music. The tempo may be adjusted based on the severity of the condition. The environment perception module 204 may play a corresponding warning phrase for a specific danger and increase the volume of the music being played to better attract the user's attention. The value of the volume may be increased based on the noise level in the cabin of the vehicle, which may be detected by a microphone. The warning phrase is played loudly to ensure that the user hears the corresponding alarm.

[0098] Furthermore, the disclosed music generation and playback process allows for many dimensions of data to be represented in a compact form by mapping environmental data onto variable musical dimensions of a musical composition. Musical dimensions include volume (loudness), pitch, rhythm, and harmony. Since phrases are melody-harmony-rhythm constructs, the melody, harmony, and rhythm musical dimensions can be used to indicate other attributes of a PoI. Composers and / or designers can Figure 2The context sensing module 204 of the system or other context sensing modules disclosed herein can be programmed to automatically change these dimensions for the phrase based on GIS data, road conditions, status information, and / or other context data. The module can also be programmed to change the phrase based on stored user preferences. Some user preferences can include the type, style, and / or genre of music of interest, the type of PoI to be detected, the safety condition to be detected, the status information to be reported, etc.

[0099] pitch

[0100] The environmental perception module 204 can adjust the pitch. For example, the note A above middle C played on a piano is considered to be the same pitch as a pure tone of 440 Hz. Music theorists sometimes use a digital scale based on the logarithm of the fundamental frequency to represent the pitch. As an example, the environmental perception module 204 can adopt the Musical Instrument Digital Interface (MIDI) standard to map the fundamental frequency f to a real number p of the pitch, as shown below: p = 69 + 12*x [log2 (f / 440)], where x is a real value from the data being monitored for possible hazards. Different environmental data values ​​of x result in different pitches being played and heard. This is one of many algorithmic examples that can be used to convert a data set into music. In one embodiment, the data (or corresponding function representation) is not used to generate a musical score, such as playing a note at the location of each peak, because a pleasant sound cannot usually be generated in this way.

[0101] In one embodiment, the environment perception module 204 converts digital sequences from various environmental data sources (such as databases, sensors, control modules, etc.) into waveforms of musical sounds. In another embodiment, the environment perception module 204 does not directly convert digital sequences into musical sound waveforms, but obtains the musical form of a specific PoI and plays each note (of a specific frequency) within a specific short period of time. Musical form refers to the structure of a musical work or performance. Multiple organizational elements can determine the formal structure of a piece of music, such as the arrangement of musical units showing repeated or changing rhythms, melodies and / or harmonies. This can include factors such as the arrangement of musical instruments or the arrangement of symphonic works. These organizational elements can be broken down into smaller units, called phrases.

[0102] Clicks are avoided by arranging the waveform to cross zero at the beginning and end of each note. In this type of arrangement with nested sequences, both repetitive and random musical forms tend to produce pleasing tones. The environment sensing module 204 can determine the frequency of the note directly from the value of the element or from the cumulative sum of each value (such as the sum of one or more distance values, temperature values, humidity values, and / or other parameters).

[0103] Rhythm

[0104] The environmental perception module 204 can also adjust the tempo based on the environmental data. Tempo is the speed or pace of a given piece of music, usually indicated by beats per minute (BPM). This means that note values ​​(e.g., quarter notes or crotches) are specified as beats, and corresponding markings indicate that a certain number of these beats are to be played per minute. The greater the tempo, the greater the number of beats played in one minute, and the faster the music is played. The environmental perception module 204 can allow the user to set the initial tempo and / or other tempos of the music. This can be an adjustable parameter set via the display 212 or other interface. In one embodiment, tempo is used to indicate the distance to and from a PoI. Tempo is an exemplary data dimension, and other data dimensions are disclosed herein that can be used to indicate various environmental information.

[0105] harmony

[0106] The environment perception module 204 can adjust the harmony of the music to indicate different environmental data. Harmony is a combination of notes that sound at the same time to produce chords and chord progressions, with a pleasing effect. Harmony may include pitch simultaneity, and therefore the chord is actual or implied. There is a specific basic harmony. The basic chord consists of three notes: the fundamental, the third note above the fundamental, and the fifth note above the fundamental (which is a minor third note on the third note above the fundamental). For example, in a C chord, the notes are C, E, and G. In an A flat chord, the notes are A♭, C, and E♭. In many types of music, especially Baroque and jazz, the basic chord is often strengthened by "extended tones". Extended tones are tone classes that hit dissonant intervals in a given key. The most basic and common example of extended tones is the "seventh" (minor, or flat seventh) - so named because it is the seventh degree of the scale in a given key. While the actual degree is a flattened seventh, the name is simply "seventh." So, in a C7 chord, the notes are C, E, G, and B♭. Other common dissonant extensions include the ninth, eleventh, and thirteenth. In jazz, chords become very complex, with several extensions.

[0107] The environmental awareness module 204 may play a composition including extended sounds. For example, when a hazard is detected, an extended sound may be played. When the hazard is not implemented and / or no longer exists, the environmental awareness module 204 may "resolve" the extended sound. For example, when a vehicle is about to experience a location where a lake affects snow accumulation, a dissonant extended sound may be produced, and after passing the location, the dissonant sound may disappear. Dissonant chords (chords with extended sounds) may be played and resolved by playing consonant chords. Good harmony generally sounds pleasant when there is a balance between consonant sounds and dissonant sounds. Simply put, good harmony is produced when there is a balance between "tense" and "relaxed" moments. For this reason, extended sounds are "prepared" and then "resolved."

[0108] Preparing a stretch means laying down a series of consonant chords that smoothly lead to dissonant chords. This way, the stretch section is built up smoothly without disturbing the listener. Once the piece reaches the secondary climax, the listener is given a moment of relaxation to relieve the tension, which is achieved by playing consonant chords that resolve the stretch of one or more previous chords. The relief of tension usually makes the listener happy.

[0109] Extended sounds may be generated and used as a path to the PoI in a musically perceptible manner. The more extended or discordant sounds, the more PoIs may not match the driver's wishes. The disclosed system harmonizes several data feeds / digital sequences by selecting a key for the music and converting the data feed / sequence to the same key if the data is similar to the monitored data, or to a discordant key if the data does not match the monitored data.

[0110] The environment awareness module 204 may indicate the presence of the PoI in the form of a musical discord or extended sound provided over the background music being played. The music PoI added to the background music is distinguishable from the background music to the driver, and the driver is able to immediately recognize the indication of the PoI it is targeting and then take appropriate action. The music PoI blends with the background music and is distinguishable to the listener, but due to the blending, the user cannot detect that the music PoI is not part of the overall music being played and is therefore indistinguishable.

[0111] The closeness of the match of the environmental data to the predefined PoI can be indicated to the user through the use of consonant and dissonant harmonies and / or other variations. The closer the match, the more perfect or consonant the harmony. In one embodiment, a different pitch is assigned to each PoI. In another embodiment, each PoI is mapped to a different instrument, and the harmony is used to process a group of multiple PoIs in a way that is pleasing to the human ear. In still another embodiment, extended sounds are used as a path to reach the PoI in a musically perceptible manner. The more extended sounds or dissonant sounds there are, the less the music PoI matches the user's preferences for music. Other variations may include, for example, a specific drum line, the addition of a melody, a pitch change, a temperature change, the sound of a cowbell, a transition from one genre to another, etc.

[0112] The environment awareness module 204 may allow the user to configure the style, instrument, pitch mapping, and time control of the music (also referred to as the environment reporting music). Thus, the user is able to select music settings that the user can relate to. The user may select the music style, instrument, one or more keys of the music, the tempo, and the type of harmony to be played. The environment awareness module 204 may coordinate several data feeds / digital sequences from one or more environmental data sources and select a key for the music. The key may be one of the selected keys if the received data is similar to or matches the data associated with the PoI (or other environmental information being tracked), or the key may be a discordant key if the received data does not match the data associated with the PoI (or other environmental information being tracked).

[0113] Various software and hardware techniques may be implemented by the environment perception module 204 and used to generate music based on environmental data and / or information. The user may select any combination of musical style, instrument (instrument selection), pitch mapping, time control, and / or harmony to customize the feel of the music being generated. The environment perception module 204 may display a screen via which the user may make and adjust these selections. In addition, the environment perception module 204 may include software for creating new musical forms from an initial set of different musical forms previously created or initially selected to indicate specific PoIs and / or other environmental information.

[0114] Techniques may include artificial neural networks (trained), cellular automata, generative grammars, genetic algorithms, self-similarity (fractal music), transition networks, and mapping strategies. Artificial neural networks (trained) may be used to generate surprising moves that conform to the underlying corpus used to train them - imitating the style of that corpus. Neural networks may be trained based on received environmental data and performed actions. Cellular automata may be used to implement unique personal combination strategies. Generative grammars may be used to enable contextual and hierarchical organization of musical material in a sequential manner. Genetic algorithms may be used to generate a large number of small musical form fragments for use in process-like compositions. Self-similarity may be used to generate new music through self-similarity transformations. Transition networks may be used to enable different combination strategies in the sense of programming languages. Mapping strategies include decisions about mapping GIS and / or other environmental data to musical forms in a manner suitable for encoding by a programming language.

[0115] For example, the context-aware module 204 may start with a musical form that has been designated as indicating a city, and select and apply one or more of the above-described methods and techniques to automatically create different and distinct musical phrases. This may be done for different cities without having to create different musical phrases for each city in advance. Similarly, the context-aware module 204 may start with a musical phrase indicating a generic store, and proceed to derive or create musical phrases for different types of stores therefrom (applying the above-described methods and techniques).

[0116] The following examples may be created by a composer and stored in the memory 210 and implemented by the environment perception module 204. The examples may be modified by the environment perception module 204 automatically and / or based on user input. The following examples include tables for mapping environmental data to various musical forms. The mapping information is used to programmatically generate music. For example, an artificial neural network can be used to generate music in a specific genre and style; after the artificial neural network is trained. The genre may be a string quartet in the style of a specific composer (such as Mozart), or a jazz band in the style of a specific singer or conductor such as (Cabe Calloway). The exemplary mappings disclosed herein are not exhaustive. Musical forms may include sonatas - with introductions, elaborations, developments, recapitulations - to describe a long journey, with PoIs, destinations, weather, road conditions, etc. Techniques such as inversions and variations may also be included. For example, a theme may be used to represent a category of cities - and then change it to indicate a specific city - thereby establishing relationships between various objects of the same kind (i.e., types and categories).

[0117] Table 1 shows an example of mapping geographic information to a musical form. In Table 1, checking the volume box indicates that the volume is increased to indicate the corresponding environmental information (Environmental Information).

[0118]

[0119] Table 1 - Example of mapping geographic information to musical form

[0120] Table 2 shows an example of mapping geographic information to musical instruments (e.g., a jazz quintet). In Table 2, environmental information is mapped to the musical style of jazz, and more specifically to a jazz quintet. Multiple data inputs are mapped to the same instrument, thereby conveying multiple types of information on five instruments by relying on musical harmony. Words can also be used to convey information if human voices are used, such as in opera or rap music genres - in this case, the environment that the user is experiencing (or the "world" around the user) is being played to the user in the form of instrumental music and singing (or human voice).

[0121]

[0122] Table 2 - Example of mapping geographic information to musical instruments

[0123] Table 3 shows an example of using chimes and chords to map driver directions. A chord consists of 3 or more notes played simultaneously. Using 3 notes, 3 3 Or 27 possible driver actions. This could be in the form of a high pitch / high volume whistle, which might be considered unpleasant in the flow of music, or in the form of some other percussion instrument.

[0124]

[0125] Table 3 - Example of using chimes and chords to map driver directions

[0126] At the simplest level, the chords played may be short phrases or music such as bells introduced into the flow of music and interspersed across different instruments.

[0127] The environment perception module 204 may also use other methods, such as: a tone or chord that changes pitch (wave frequency) when the next turn in the road is approaching (i.e., the tone or chord is played in a higher octave); a tone pulse that becomes more frequent as the user approaches the next turn in the road; a chord that changes in rhythm to indicate deceleration or acceleration; using a stereo system to change the position of the sound from left to right and / or from front to back, thereby notifying the user on one side of the vehicle that the turn in the road is open; and / or other methods. The environment perception module 204 may indicate the actions and behaviors of nearby vehicles via music. This may include a set of tones / chords that identify specific or possible events and / or provide specific warnings. A set of tones / chords may be played to indicate that a vehicle is approaching too quickly or a collision may occur, and to provide side collision, front collision, or rear collision warnings. The enclosedness of the vehicle may refer to a specific operating mode. Different tones, chords, phrases, themes, etc. may be played for different modes. A side collision warning may include using a stereo system to move the sound from left to right to indicate the direction and speed of a possible collision. Rear collision warnings may include using a stereo system to inform a user, for example, of the vehicle's speed prior to a rear collision by moving sound from rear speakers to front speakers.

[0128] A specific song may be played for the user, and the song may be changed to include different pieces that indicate changes in the environment. The specific song may be in a specific music style, genre, etc. that the user likes. The song may sound similar to a song by a specific artist that the user has indicated as a preferred artist. The song may be similar to songs in the user's playlist. The playing of music allows the user to effectively listen to the environment and detect changes in the environment through the music.

[0129] In one embodiment, a baseline of default works is stored in memory 210 and updated and / or modified over time. Each of the works as further described below may be modified based on environmental information received in real time (i.e., as environmental information is received and processed). As an example, a user of network device 200 may have visual acuity issues and instruct environmental awareness module 204 to operate in an environmental awareness mode that includes playing music that indicates environmental information. The user may be: riding a bicycle; in an autonomous vehicle; walking in a building, city, or park; driving a vehicle; or riding a bus, and perceive the surrounding environment through music. For example, the user may be aware of: an impending emergency; weather conditions; the corresponding vehicle is leaving the city or approaching a destination; and so on.

[0130] Figure 3A vehicle entertainment access system 300 is shown that includes a vehicle 301, a distributed communication system 302, and a central office 304 of a service provider. The vehicle 301 includes an environment-aware system, a portion of which is shown and labeled 306. The environment-aware system includes an environment-aware module 308 that executes an environment-aware application 310. The environment-aware module 308 communicates with a mobile device 312 and the central office 304 via a telematics module 316.

[0131] The mobile device 312 may be implemented as a display, a portable network device, and / or other network device. The display (or viewing device) may include a head-up display (HUD), a laser holographic device, a liquid crystal display (LCD), and / or other 2D and 3D displays. The windshield and / or windows may be used to display items and may therefore be referred to as displays. The telematics module 316 includes a transceiver 318 and a telematics control module 320, which may include a Bluetooth® transceiver, a wireless local area network (WLAN) transceiver, a wireless fidelity (Wi-Fi®) transceiver, a cellular transceiver, and / or other transceivers.

[0132] The environment perception module 308 is connected to the telematics module 316, the vehicle control module 321, and other modules 324 via a bus 322 (e.g., a controller area network (CAN) bus). The environment perception module 308 stores occupant information 330 and preferences 331 and environment information (or data) 333 in a memory 332. As used herein, the term "occupant" may be replaced by the term "user". As further disclosed herein, the environment perception module 308 sends environment information and / or works to be played to the mobile device 312 based on the occupant information and preferences.

[0133] The environment awareness module 308 communicates with a monitoring station 340 of the central office 304 via the distributed communication system 302. The monitoring station 340 may include a control module 342, a transceiver 344, and a database and / or memory 346. Occupant information and preferences may be stored in the database and / or memory 346 and shared with the environment awareness module 308. The monitoring station 340 may operate as, include, and / or access a cloud-based storage device. The environment awareness module 308 may access the Internet and provide an Internet connection for downloading advertisements, educational information, business and / or building identification information, restaurant information, etc. via the distributed communication system 302. The environment awareness module 308 may convert the provided information into music and play the music via the vehicle's audio system, or transmit the information and / or composition to the mobile device 312 for playback via the mobile device 312.

[0134] In one embodiment, the context-aware application 310 is implemented as part of a vehicle communication interface module (e.g., an OnStar® module), and the monitoring station 340 provides remote services associated with the vehicle communication interface module. The vehicle communication interface module may be one of the other modules 324. The occupant information and information related to the object may be obtained through an Internet connection and / or a backend service provided within the vehicle 301, as further described below. The backend service may be provided at least in part by the monitoring station 140 and / or other remote network devices.

[0135] The environment sensing module 308 can detect, receive, determine and / or obtain various environmental information and indicate the environmental information to the mobile device 312 in the form of music. The environmental information can be broadcast to the mobile devices 312. Each mobile device 312 can receive the data fed by the environment sensing module 308 and convert the data into music via the environment sensing module 350 based on the preferences of the specific user of the mobile device. The environment sensing module 308 and the environment sensing module 350 can communicate with each other. Figure 2 The environment perception module 204 operates similarly.

[0136] Figure 4 A vehicle 301 is shown incorporating vehicle systems 400 including an environment interaction system 306. The vehicle 301 includes a vehicle control module 321, which may include an environment perception module 308, or the environment perception module 308 may be included as part of an infotainment module 402. The vehicle 301 also includes a telematics module 316, a memory 332, a navigation module 404, sensors 406, a microphone 407, a propulsion system 408, an inertial measurement module 410, and other vehicle systems 412.

[0137] The telematics module 316 provides wireless communication services within the vehicle 301 and communicates wirelessly with a service provider. The telematics module 316 may support Wi-Fi®, Bluetooth®, Bluetooth Low Energy (BLE), Near Field Communication (NFC), Cellular, Legacy (LG) Transmission Control Protocol (TCP), Long Term Evolution (LTE), and / or other wireless communications, and / or operate according to Wi-Fi®, Bluetooth®, BLE, NFC, Cellular, and / or other wireless communication protocols. The telematics module 316 may include a transceiver 318 and a telematics control module 320. The telematics control module 320 may include a global positioning system (GPS) 414. The transceiver 318 includes a physical layer (PHY) module 416 and a media access control (MAC) module 418. The PHY module 416 communicates wirelessly with network devices inside and outside the vehicle 301. The MAC module 418 may perform pattern recognition, channel addressing, channel access control, and filtering operations.

[0138] The infotainment module 402 may include and / or be connected to an audio system 420 and / or a video system 422 including one or more displays 424. The audio system 420 and the video system 422 may provide vehicle status information, diagnostic information, predictive information, entertainment features, and other vehicle and non-vehicle related information. The entertainment features may include: object information; weather information, commercial information; and / or other environmental information. The display 424 may also display (i) items captured by the external camera 430, including detected objects outside the vehicle, and (ii) information associated with the detected objects. The audio system 420 may play music indicating the presence and / or information associated with the detected objects. By providing environmental information through music, the driver of the vehicle 301 is better able to keep his or her eyes on the road ahead and is less likely to be distracted by visually provided indicators. Audio indications may be provided in place of or in addition to visual indications.

[0139] The vehicle control module 321 can control the operation of the vehicle 301, including the operation of the propulsion system 408 and other vehicle systems 412. The memory 332 can store occupant information 330, occupant preferences 331, entertainment information 423, Internet-based information 425, object information 426, business information 427, map data 428, and other environmental data 429. The occupant (or personal) information 330 may include, for example, the occupant's name, occupant identifier (ID), occupant password, facial recognition information, fingerprint recognition data, eye recognition data, and other personal information. The object information 426 and business information 427 may include the name of the object and business, the address of the business, the age of the object and business, the history of the object and business, etc. The occupant preferences 331 may include personal applications, preferred podcasts, music preferences, types of information searched, online videos watched, social media information, preferred restaurants and / or bars, hobbies, work-related information, vacation locations, items searched and / or purchased, preferred hotels, preferred airlines, preferred travel methods, etc. Restaurant-related information may include preferred food types, preferred coffee shops, ratings and cost preferences, etc.

[0140] Entertainment information 423 may include some of the following: occupant preferences, and / or trivia information and / or information related to the local environment surrounding vehicle 301, local events, popular "things to see" in the local area, Internet-based information 425, object information 426, commercial information 427, etc. Internet-based information 425 may include information related to websites, products, services, videos, restaurants, and / or other items searched and / or viewed via the Internet.

[0141] Object information 426 may include animal and / or plant information, architectural information, historical information, landmark information, building information, etc. Business information 427 may include historical information, information describing the business, building information, architectural information, stock information, company status information, etc. Map data 428 may include geographic data, location identification information, and local weather. Other data 429 may include traffic information, time of day information, etc.

[0142] The navigation module 404 executes the navigation application to provide navigation services. As an example, if the vehicle 301 is an autonomous driving vehicle, the navigation module 404 can guide the vehicle control module 321 to a specific location.

[0143] Sensors 406 may include cameras 430 and other vehicle sensors 434 (e.g., vehicle speed sensors, temperature sensors, pressure sensors, flow rate sensors, etc.). Camera 430 may be a 2D camera, a 3D camera, a depth camera, an infrared camera, a visible light camera, and / or other suitable cameras. Camera 430 and telematics control module 316 may be used to detect objects outside vehicle 301. Camera 430 may include a corresponding infrared (IR) source 436. When, for example, there is low visibility (low visibility distance), such as in foggy weather, an IR source may be used on the outside of the vehicle. In one embodiment, a visible light camera and an IR camera are used. Camera 430 is used to monitor the area around the vehicle and may be located at various locations on the vehicle. Microphone 407 may be used to receive audio commands and / or receive verbal input from an occupant and / or detect noise levels within the cabin of vehicle 301.

[0144] Propulsion system 408 may include internal combustion engine 444, transmission 445, and / or one or more electric motors 446 to propel vehicle 301. Vehicle control module 321 may control propulsion system 408 to cause vehicle 301 to move to a selected location.

[0145] Inertial measurement module 410 includes inertial measurement sensor 448, such as accelerometer and / or rate sensor. As an example, inertial measurement module 410 may be located at or near the center of gravity (cg) point of the vehicle. Inertial measurement sensor 448 may be used to measure longitudinal and lateral acceleration and / or longitudinal and lateral velocity of vehicle 301. Longitudinal and lateral acceleration and longitudinal and lateral velocity may be integrated to determine the pitch angle and roll angle of vehicle 301, thereby determining the orientation of the vehicle.

[0146] The environmental awareness module 308 may detect environmental information related to various types of information stored in the memory 332 and indicate the environmental information through music via the audio system 420. The environmental awareness module 308 may also indicate information collected from the navigation module 404, the sensor 406, the inertial measurement module 410, the propulsion system 408, and other vehicle systems 412. The other vehicle systems 412 may include an air conditioning system 450, a seat system 452, a lighting system 454, and / or other vehicle systems.

[0147] Figure 5 An environmental notification system 500 is shown that provides environmental data directly to a mobile device. The environmental notification system 500 may include a central office 502, a distributed communication system 504, and a mobile device 506. The central office 502 may be similar to Figure 3 The mobile device 506 may be connected to the central office 304 and include a monitoring station 510. The monitoring station 510 may include a control module 512, a transceiver 514, and a database and / or memory 516. Figure 3 The mobile device 506 is configured and operates similarly to the mobile device 312 of FIG. Figure 2-3 The environmental awareness module 520 operates similarly to the environmental awareness modules 204 and 350 .

[0148] Figure 6 An environment perception module 600 and a memory 602 are shown. The environment perception module 600 may represent any of the other environment perception modules disclosed herein, including Figure 2-5 The environment perception module 600 may include a variable piece data extraction module 610, a piece compiler module 612, a piece configuration module 614, a mapping module 616, and one or more interfaces (two interfaces 618, 620 are shown). In one embodiment, the interface includes one or more buses connected to other modules, sensors, memories, transceivers, and / or interfaces (e.g., touch screens, keypads, keyboards, microphones, etc.). The memory 602 may store variable music and encoded piece templates 622, environmental data 624, data / piece mappings 626, user preferences 628, user-selected environmental information, and / or conditions of monitors 629. The environmental conditions may be exterior or interior of a vehicle of which the user is an occupant.

[0149] The variable composition data extraction module 610 applies one of the data / composition mappings 626 to convert the ambient data into the format required by the composition compiler module 612 to generate a music signal for playback via an audio system 630, such as one of the audio systems described above.

[0150] The work configuration module 614 receives input from the composer via interface 618. The composer can upload a variable and coded music work template, create a variable and coded music work template and / or modify an existing variable and coded music work template. One or more of the stored variable music work templates 622 can be uploaded via one or more composers. The variable and coded music work template refers to a combination of elements that can be intelligently modified in a music work and the work, and these elements can be changed based on an environmental data source. For example, a variable and coded music work template may include one or more works with variables that can be changed based on received environmental data values. These variables may indicate and / or be used to change any musical characteristics described herein, such as rhythm, pitch, harmony, tone, volume, voice, melody, dissonance, harmony, theme, etc. The user can select one or more of the variable and coded music templates, and the work compiler module 612 creates music based on the variable and coded music template. Environmental data may include any data representing certain environmental information and / or PoI that the user is interested in hearing.

[0151] The user may input the context information of interest via interface 620. The user may also request that different compositions and / or tracks be played. The user may also input user preferences 628, which may include any of the user preferences described herein, including music genre (e.g., jazz, blues, rhythm, rock, country, soul, dance, etc.), tempo, minimum and / or peak volume, and / or other music preferences. The user may also indicate the types of music that the user does not prefer. Figure 6 As shown, preferences 628 may include user-selected environmental information and / or conditions to be monitored. Audio tracks may refer to music that is played in real time for the user via audio system 630.

[0152] Mapping module 616, although shown as separate from work configuration module 614, can be integrated as part of work configuration module 614. Mapping module 616 generates data to work mapping, based on the environmental data elements, can modify and set variables in the variable and encoded music work template. Work compiler module 612 uses the selected variable and encoded music template and applies the variable data extracted from the environment to create music. Variable work data extraction module 610 can map the received environmental data and / or the detected specific environmental conditions to a specific work. Baseline (or default) works can be selected for standard (or default) context sets, and / or custom works can be created by the user and selected and used for standard context sets or custom (or user-created) context sets.

[0153] The user may select any combination of various parameters to customize the feel of the music being generated, such as musical style, instrumentation (instrument selection), pitch mapping, time control, harmony, etc. Certain types of compositions are more suitable for certain types of situations. For example, when tracking the distance between a user (or a user's vehicle) and an object (e.g., another vehicle), a continuously changing melody may be appropriate as opposed to occasional dings. For example, the melody may increase in tempo as the user (or the user's vehicle) gets closer to the object. This is substantially different from how drivers traditionally detect oncoming vehicles via vision and using vibrations in mirrors, displays, and / or seats, for example.

[0154] The mutable and encoded music templates can be simple (unencoded) or complex (encoded). Below is an example of a specific implementation of an encoded template. If the sun intensity has a value of 0-10, each sun intensity (or sun intensity integer) value can be associated with a specific range of pitch values. For example, if the sun intensity is a first value, then a pitch value of 11-40 is used. If the sun intensity value is a second value, then a pitch value of 41-100 is used. Then, at the composition compiler module 612, the environmental data can then be applied to the template and the actual musical instructions for generating the music are created.

[0155] The following will be relative to Figure 7 The method further describes the operation of modules 600, 610, 612, 614 and 616. Figure 7 The method is taken as an example, and other similar methods can be implemented.

[0156] Although the following method is mainly relative to Figure 6 The method is applicable to Figure 2-5 Embodiment of the invention. Figure 7 Shows the operation Figure 6 The method may be performed iteratively. The method may start at 700. At 702, the work configuration module 614 may receive a mutable and encoded music work template and store it in the memory 602.

[0157] At 704, the work configuration module 614 can receive and store user preferences, user-selected conditions to be monitored, user-selected environmental information to be tracked and / or monitored, etc. As a few examples, the user may wish to know: the distance to a particular destination, when the user is approaching a particular destination, an impending change in weather, whether a particular type of restaurant is close, the status of flaps on an airplane the user is flying, whether a hazardous condition exists, etc.

[0158] At 706, the variable work data extraction module 610 receives environmental data from one or more sources, at least some of which are described above. The variable work data extraction module 610 can filter out data that is not relevant to the environmental information that the user requested to monitor and / or other conditions being detected. In one embodiment, in addition to the information and / or conditions that the user requested to monitor, the environmental perception module 600 monitors a set of default conditions.

[0159] At 708, the variable work data extraction module 610 converts the environmental data to a format for the work compiler module 612 based on the data-to-work mapping received from the mapping module 616. Custom programming of the conversion and / or mapping can be implemented in software. As an example, the generated music can change as the GIS data changes in time and space. At 710, the work compiler module 612 generates an audio signal that is played as music by the audio system 630 for the user to listen to and hear the surrounding environment aspects reported to the user in the form of music. Initially, the work compiler module 612 can generate audio output based on a default or initial variable and encoded music work template. This can include automatic generation of musical forms and canned musical forms that reuse known data sources from pre-recorded music.

[0160] At 712, the work configuration module 614 may determine whether a request to change the stored preferences has been received from the user. This may include updating, adding, and / or removing preferences. If so, operation 714 may be performed, otherwise operation 716 may be performed. At 714, the work configuration module 614 may receive and store the preference changes.

[0161] At 716, the work configuration module 614 can determine whether a request to update, remove, and / or add monitored conditions has been received from a user. If so, operation 718 can be performed. At 718, the update changes can be received and stored.

[0162] At 720, the composition configuration module 614 may determine whether the user has requested a change to the currently playing track. If not, operation 722 may be performed. If a change to the current track is requested, operation 724 may be performed. At 722, the composition configuration module 614 adjusts the current mutable and encoded composition template based on the user preferences and conditions to be monitored.

[0163] At 724, the composition configuration module 614 may proceed to operation 726 if not operating in the manual selection mode, and if operating in the manual selection mode, the module may proceed to operation 728. At 726, the composition configuration module 614 may automatically change the mutable and encoded composition template based on the user preferences and the conditions to be monitored, and then provide it to the composition compiler module 612. At 728, the composition configuration module 614 allows the user to select and / or adjust (or customize) the mutable and encoded composition template, and then provide it to the composition compiler module 612.

[0164] At 730 , the variable work data extraction module 610 continues to receive environmental data from various sources. Operation 708 may be performed after operation 730 .

[0165] The above operations are intended as illustrative examples. Depending on the application, the operations may be performed sequentially, synchronously, simultaneously, continuously, during overlapping time periods, or in a different order. Moreover, depending on the implementation and / or order of events, any operation may not be performed or skipped.

[0166] The above examples provide audio as a way for users to identify aspects of the environment. Examples include representing attributes, features, and conditions associated with the environment surrounding the user as music. This is particularly helpful for visually impaired users. A large amount of environmental data can be provided in an easily recognizable format through music. This allows the user to receive environmental information without the user having to divert their sight from, for example, the road ahead, thereby improving the safety of operating a vehicle and / or other forms of transportation (e.g., a bicycle). This is done without the need for a display or screen. These examples allow the user to select attributes of interest. These examples also allow the environmental perception module involved and / or the user to assign music templates and / or works to the attributes of interest.

[0167] The above examples allow the user to detect various conditions via music. As a few examples, the above system may allow the user to detect: the user or the corresponding vehicle is approaching black ice; it is about to rain; there is a traffic jam ahead; the vehicle in which the occupant is currently riding is leaving the city limits; and so on. The system may automatically select or the user may select one or more pieces and / or tracks of music for each condition. Different instrument sounds may be used for different occasions. As an example, the playing of wind chimes and the volume and / or frequency of the generated sound may be used to represent the strength and speed of the wind. The music may be changed to track the presence of inanimate objects (e.g., vehicles and non-vehicle objects) as well as animate objects (e.g., people and animals).

[0168] Examples allow a user to listen to the surrounding environment or "world" around the user in the form of music having various combined elements representing certain aspects of the environment. At least some of the combined elements may represent aspects that the user is interested in hearing. These examples may be implemented while the user is walking on a sidewalk, riding public transportation, renting a car, operating some form of transportation, etc. The music changes as the surrounding environment changes.

[0169] Examples may include limiting the number of aspects and / or conditions at the indicated moment and / or the number of different and simultaneously played works and / or changed audio characteristics. As an example, the number of aspects and / or conditions may be limited to 3-5 and / or the number of different and simultaneously played works and / or changed audio characteristics may be limited to 3-5 to ensure that the user can detect, identify and, if necessary, react to one or more aspects and / or conditions indicated. A single work may be played to indicate multiple aspects and / or conditions. Audio characteristics may include melody, pitch, harmony, volume, rhythm, etc. The phrases played may be constructed from images, themes, and units.

[0170] Examples include converting environmental data into music, rather than simply detecting whether a condition exists and then creating an auditory sound. For example, a vehicle that detects the proximity (or distance) of an object while backing up and then emits a beep to indicate that the distance between the vehicle and the object is decreasing is different from a vehicle that converts distance data into music that includes a melody that changes to indicate that the object is getting closer. As another example, the disclosed system can be used to indicate to the operator of a first vehicle via music that a second vehicle is in a blind spot. The vehicle operator is able to detect the presence of the second vehicle and the approximate distance of the second vehicle approaching the first vehicle by simply listening to the music and not turning and / or tilting the operator's head to see the first vehicle, looking in the mirror, and / or looking at the display of the first vehicle. The operator may be listening to a song, and the song changes to indicate the approaching second vehicle. For example, the song may include a "whoosh" sound (or a conflict of symbols) as part of the music, and the whoosh sound may become louder as the second vehicle gets closer. The song played allows the operator to understand the surrounding environment with minimal interference. The music played is designed to adapt to the real-time perception of changes in the surrounding environment and plays continuously when the system is enabled. Certain aspects may be played continuously (such as the change in distance to a destination), while other aspects are played as part of a song during corresponding time periods. Music may be played as background music while allowing the user to hear, see and / or feel other traditional vehicle indicators.

[0171] Examples disclosed herein help reduce the cognitive load on a driver or pilot by providing auditory information and allowing for a reduction in the amount of visual information.

[0172] Figure 8An exemplary diagram 800 is shown, which shows features associated with a musical representation of environmental data 802 of the above system, which can be implemented by any of the above environmental perception modules. The generated music can represent distance as a scale represented by line 804, speed as a rhythmic change represented by line 806, and changes in conditions as tones represented by line 808. The music can include customization (represented by line 812) and / or be generated using various types of techniques (represented by line 814). The music can use different musical vocabulary genres represented by line 816 to represent different aspects and conditions. Different aspects and conditions can be represented as parts of the music flow, represented as line items 818. New music can be created, represented as line 820. Different musical vocabularies can be used for different PoIs (represented by 822), and different musical techniques can be used (represented by line 824).

[0173] Changes in situation 808 may include change direction alerts 830, road alerts 832, traffic alerts 834, vehicle alerts 836 (e.g., distance to nearby vehicles 838), vehicle health 840, and weather alerts 842. Customization 812 may include music genres 844 and aftermarket digital assets 846. Technology 814 may include artificial intelligence (AI) for music styles 848, electronic music 850, fuzzy logic 852, music lexicon 854, and synthesizers 856. Music vocabulary genres 816 may include classical 858, country 860, jazz 862, metal 864, and rhythm and blues 866. New music 820 may include human composer canvases 868 and automated compositions 870. Music vocabulary for PoIs may include boundaries 872, cities 874, gas stations 876, lodging 878, food 880, PoIs 882, road conditions 884, and weather conditions 886. Music techniques 824 may include amplification of sounds and note ranges 890, changes in scale 892, changes in grammar 894, melodic structures 896 including swapping melodies in a stream 897, phrase indication information 898, and removal of sounds and note ranges 899 following a beat / rhythm 888.

[0174] like Figure 6 As shown, the user interface for customizing the recipe can be expanded to include other features, such as algorithms for specific data streams so that the specific data streams are converted into musical sounds. Themes and sounds can be played based on ambient data without the need for background or ambient "songs" to play in the background.

[0175] The environment awareness module disclosed herein can be implemented in a server or a portable device. The environment awareness module can be implemented and deployed on a variety of computing platforms with different form factors. One such platform can be a telematics platform. The environment awareness module can be implemented on a telematics module (such as a Figure 3-4 The telematics module shown is implemented in the embodiment of the present invention.

[0176] In one embodiment, instead of automatically generating music based on environmental data (or signals) and / or conditions, a database of pre-recorded musical pieces (single instrument, multi-instrument or orchestral pieces, etc.) is provided based on the detected environmental data and / or conditions. The system will then enable the user to perform a mapping of pre-recorded music to signals (source of environmental data).

[0177] The above examples may include more complex musical forms to indicate spatial relationships and / or other relationships between PoIs. Some exemplary relationships include connection relationships, adjacency relationships, inclusion relationships, overlapping relationships, proximity relationships, coincidence relationships, elevation relationships, and parallel relationships. An example of a connection relationship is that Water Street is connected to 18th Avenue. An example of an adjacency relationship is that a city park is adjacent to a university. An example of a inclusion relationship is that a building footprint is contained within a lot boundary. An example of an overlapping relationship is that a railroad crosses a highway. An example of an adjacency relationship is that a courthouse is near the state capitol. An example of a coincidence relationship is that a city park is consistent with a historic site polygon. An example of an elevation relationship is that the state capitol is uphill from the water. An example of a parallel relationship is that a bus route follows along a street network.

[0178] Such examples may include playing special customizable musical chimes or phrases to indicate left turn, right turn, sharp left turn, sharp right turn, stop immediately, stop sign ahead, school crossing ahead, railroad crossing ahead, give way, other types of interactions ahead, and roundabouts ahead and present to take based on the trip map. The environmental perception module may execute training games to teach people to learn which sounds (musical phrases) correspond to which rules and which locations. Musical representation of structured and unstructured data enables GIS and other environmental information to be musically represented in such a way that the cognitive load on the driver is reduced in a pleasing manner.

[0179] The foregoing description is merely illustrative in nature and is in no way intended to limit the present disclosure, its application or use. The broad teachings of the present disclosure can be implemented in a variety of forms. Therefore, although the present disclosure includes specific examples, the true scope of the present disclosure should not be so limited, because other modifications will become apparent by studying the drawings, the specification and the appended claims. It should be understood that one or more steps within the method can be performed in a different order (or simultaneously) without changing the principles of the present disclosure. In addition, although each of the embodiments is described above as having certain features, any one or more of those features described with respect to any embodiment of the present disclosure may be implemented in the features of any other embodiment and / or combined with the features of any other embodiment, even if the combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the permutation of one or more embodiments with each other remains within the scope of the present disclosure.

[0180] Spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "engaged," "coupled," "adjacent," "next to," "on top," "above," "below," and "disposed." Unless explicitly described as "directly," when describing a relationship between a first element and a second element in the above disclosure, the relationship can be a direct relationship in which no other intervening elements exist between the first element and the second element, but can also be an indirect relationship in which one or more intervening elements exist between the first element and the second element (spatially or functionally). As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A or B or C) using a non-exclusive logical or, and should not be construed to mean "at least one of A, at least one of B, and at least one of C."

[0181] In a diagram, the direction of an arrow (as indicated by the arrow head) generally shows the flow of information (such as data or instructions) of interest to the illustration. For example, an arrow may point from element A to element B when element A and element B exchange various information but the information transmitted from element A to element B is relevant to the illustration. This one-way arrow does not mean that no other information is transmitted from element B to element A. In addition, for information sent from element A to element B, element B may send a request or receipt confirmation to element A for the information.

[0182] In this application, including the following definitions, the term "module" or the term "controller" may be replaced with the term "circuit". The term "module" may refer to, be part of, or include: an application specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip.

[0183] A module may include one or more interface circuits. In some examples, the interface circuit may include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module of the present disclosure may be distributed among multiple modules connected via interface circuits. For example, multiple modules may allow for load balancing. In another example, a server (also referred to as a remote or cloud) module may perform certain functions on behalf of a client module.

[0184] As described above, the term code may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or objects. The term shared processor circuit includes a single processor circuit that executes some or all of the code from multiple modules. The term group processor circuit includes a processor circuit that executes some or all of the code from one or more modules in combination with additional processor circuits. References to multiple processor circuits include multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term shared memory circuit includes a single memory circuit that stores some or all of the code from multiple modules. The term group memory circuit includes a memory circuit that stores some or all of the code from one or more modules in combination with additional memory.

[0185] The term memory circuit is a subset of the term computer-readable medium. As used herein, the term computer-readable medium does not include transient electrical or electromagnetic signals propagated through a medium (such as on a carrier wave); therefore, the term computer-readable medium may be considered to be tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0186] The apparatus and methods described in this application may be implemented in part or in whole by a special purpose computer created by configuring a general purpose computer to perform one or more specific functions embodied in a computer program. The above-mentioned function blocks, flow chart components and other elements are used as software specifications, which can be converted into computer programs by routine work of skilled technicians or programmers.

[0187] The computer program includes processor executable instructions stored on at least one non-transitory tangible computer readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0188] A computer program may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated by a compiler from source code; (iv) source code executed by an interpreter, (v) source code compiled and executed by a just-in-time compiler, etc. As examples, the source code may be written using syntax from the following languages: C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, JavaScript®, HTML5 (Hypertext Markup Language, Revision 5), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SIMULINK, and Python®.

Claims

1. An environment perception system, comprising: a memory configured to store environmental data, one or more musical composition templates, and one or more mappings, wherein the environmental data indicates at least one of a state, a condition, or a change in an environment in which the environmental awareness system is located; A first module configured to receive the environmental data and store the environmental data in the memory; and The second module is configured to converting the ambient data into a music signal based on the one or more musical composition templates and the one or more mappings, including modifying variables in the one or more musical composition templates based on the ambient data, and playing a musical composition via an audio system based on the music signal to audibly indicate at least one of a state, a condition, or a change in the environment, The second module is configured to change the musical work to alter an amount of musical mismatch relative to user preferences based on proximity of the environmental data to a selected point of interest among different points of interest; wherein the different points of interest are destinations selected by the user.

2. The system of claim 1, further comprising the audio system, wherein: The audio system is configured to receive the music signal and play the music piece; and The musical work includes instrumental music.

3. The system according to claim 2, wherein: The musical work also includes a vocal part.

4. The system of claim 1, wherein: The memory is configured to store user preferences; and The second module is configured to generate or modify at least one of the one or more musical composition templates or the one or more mappings based on user preferences.

5. The system of claim 1 , further comprising an interface configured to receive user input, in, The second module is configured to select the one or more music composition templates based on the user input, based on a user preference indicated by the user input or an instruction included in the user input, thereby changing the track of the music composition.

6. The system according to claim 1, wherein: The second module is configured to generate the music signal to include different music phrases for different points of interest.

7. The system according to claim 1, wherein: The second module is configured to generate the music signal to include different music phrases for different environmental conditions.

8. The system according to claim 1, wherein: The second module is configured to generate the music signal to include a motif to musically indicate a distance to one or more points of interest.

9. The system according to claim 1, wherein: The second module is configured to generate the music signal to include different musical pieces for different instruments so as to indicate different points of interest musically.

10. The system according to claim 1, wherein: The second module is configured to generate the music signal to include sound effects to musically indicate changes in different environmental states or conditions.

11. The system according to claim 1, wherein: The second module is configured to generate the musical signal to include dissonant chords to musically indicate a hazardous condition.

12. The system of claim 1, wherein: The second module is configured to generate the music signal to include an extended sound or a discordant sound to musically indicate that a current state or condition does not match a user request.

13. The system of claim 1, wherein: The second module is configured to receive user input and adjust a parameter of the music signal based on the user input; and The parameters include at least one of a music style parameter, an instrument parameter, a pitch mapping parameter, a time control parameter, and a harmony parameter.

14. The system of claim 1, wherein: The second module is configured to adjust at least one of the rhythm, pitch, volume, melody, or harmony of the music signal based on changes in the environmental data.

15. The system of claim 1, wherein: The environmental data is received from one of one or more sensors.

16. The system of claim 1, wherein: The environmental data is received from a wireless network or a distributed network.

17. The system of claim 1, further comprising a transceiver configured to transmit the environmental data from the vehicle to a mobile network device, wherein: The memory is implemented in the vehicle; and The second module is implemented in the mobile network device.

18. The system of claim 1, further comprising a transceiver configured to transmit the music signal from the vehicle to a mobile network device, wherein: The memory and the second module are implemented in the vehicle; and The audio system is implemented in the mobile network device or is connected to the mobile network device.

19. A network device comprising: The system according to claim 1; as well as A transceiver is configured to receive the environmental data from at least one of a sensor, a vehicle, or a distributed network.

20. A vehicle comprising: The system according to claim 1; as well as An audio system configured to play music within the vehicle based on the music signal.

Images